Double-Take Availability User's Guide Printable Version

This document is a single HTML page that contains all of the separate HTML pages in the Double-Take Availability User's Guide. This single HTML document is designed to make printing the entire User's Guide easier. You can also print sections of the User's Guide by using your browser's print preview feature and printing only the desired pages. Keep in mind that the page numbers as seen in the Table of Contents may not match the page numbers when you print the document because of differences in printer configurations.

This printable document is not a standalone document. It requires the style definition files and graphic files used in the separate-page User’s Guide. Therefore, if you try to view or print this document on its own (for example, by copying or downloading the single file to a different location), the output will not have any formatting or graphics included.

Table of Contents

Welcome to the Double-Take Availability User's Guide 6
Getting started 7
Frequently used pages 8
Printable Version 9
Resources 10
Legal 11
Double-Take Availability overview 12
Core operations 14
Full-server protection 18
Configurations 19
Double-Take Availability requirements 24
Installation 29
Installation and upgrade notes 30
Installing or upgrading Double-Take Availability Linux servers 32
Installing or upgrading Double-Take Availability Windows client 34
Removing (erasing) Double-Take Availability 35
DTSetup 36
Running DTSetup 37
Setup tasks 38
Activating your server 38
Modifying security groups 39
Replication configuration 39
Configuring file system replication 40
Configuring block device replication 41
Configuring full-server protection 43
Configuring server settings 44
Configuring driver performance settings 44
Starting and stopping the daemon 47
Starting DTCL 48
Viewing documentation and troubleshooting tools 49
DTSetup menus 50
Clients 51
Replication Console 52
Using Replication Console workspaces 52
Clearing stored security credentials 53
Failover Control Center 54
Setting the frequency of Failover Control Center console refreshes 54
Data protection 56
Establishing a data connection using the automated Connection Wizard 57
Creating a replication set 59
Establishing a connection manually using the Connection Manager 61
Establishing a connection across a NAT or firewall 65
Simulating a connection 68
Establishing full-server protection 70
Protection monitoring 72
Monitoring a connection through the Replication Console 73
Log files 79
Viewing the log files through a text editor 80
Viewing the Double-Take Availability log file through the Replication Console 81
Configuring the properties of the Double-Take Availability log file 83
Double-Take Availability log messages 84
Monitoring the Linux system log 92
E-mailing system messages 107
Statistics 111
Configuring the properties of the statistics file 112
Viewing the statistics file 113
Statistics 115
SNMP 131
Configuring SNMP on your server 131
SNMP traps 132
SNMP statistics 137
Connections 146
Data queues 147
Queuing data 148
Auto-disconnect and auto-reconnect 152
Reconnecting automatically 153
Pausing and resuming target processing 155
Disconnecting a connection 156
Mirroring 157
Stopping, starting, pausing, or resuming mirroring 158
Mirroring automatically 161
Removing orphan files 163
Replication 166
Replication capabilities 167
Replication sets 171
Creating a replication set 173
Creating or modifying replication rules manually 175
Selecting a block device for replication 176
Modifying a replication set 177
Renaming and copying a replication set 177
Calculating replication set size 178
Exporting and importing a replication set 179
Deleting a replication set 180
Starting replication 181
Inserting tasks during replication 182
Verification 183
Verifying manually 184
Verifying on a schedule 186
Configuring the verification log 188
Data transmission 190
Stopping, starting, pausing, and resuming transmission 191
Scheduling data transmission 192
Limiting transmission bandwidth 198
Compressing data for transmission 200
Failover and failback 201
Configuring failover monitoring 202
WAN considerations 205
Protecting NFS shares 207
Protecting Samba shares 207
Editing failover monitoring configuration 209
Monitoring failover 210
Failing over 213
Failing back 214
Removing failover monitoring configuration 216
Restoring Double-Take Availability data 217
Server settings 220
Identifying a server 221
Licensing a server 223
Configuring server startup options 226
Configuring network communication properties for a server 228
Queuing data 230
Configuring source data processing options 233
Configuring target data processing options 235
Specifying the Double-Take Availability database storage files 237
Specifying file names for logging and statistics 239
E-mailing system messages 241
Security 244
Logging on and off 246
Evaluating Double-Take Availability 249

Welcome to the Double-Take Availability User's Guide

This online documentation is designed to help you find detailed information about Double-Take Availability. Use the Contents pane on the left to explore the different sections or use Search or Index to find specific topics. Below are some of the most useful and most accessed topics in the online documentation.

If you are using Internet Explorer and are getting an active content blocker pop-up in your browser, you need to enable active content to run in files on your computer. From Internet Explorer, select Tools, Internet Options. On the Advanced tab, enable Allow active content to run in files on My Computer.

Getting started

Frequently used pages

Printable Version

A printable version, which is a single HTML page containing all of the separate HTML pages in the Double-Take Availability User’s Guide, is available. This single HTML document is designed to make printing the entire User’s Guide easier.

Resources

Legal 

Double-Take, Balance, Double-Take Availability, Double-Take Backup, Double-Take Cargo, Double-Take Flex, Double-Take for Hyper-V, Double-Take for Linux, Double-Take Move, Double-Take ShadowCaster, Double-Take for Virtual Systems, GeoCluster, Livewire, netBoot/i, NSI, sanFly, TimeData, TimeSpring, winBoot/i and associated logos are registered trademarks or trademarks of Double-Take Software, Inc. and/or its affiliates and subsidiaries in the United States and/or other countries. Microsoft, Hyper-V, Windows, and the Windows logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. Red Hat is a registered trademark of Red Hat, Inc. All other trademarks are the property of their respective companies.

© 1996-2010 Double-Take Software, Inc. All rights reserved.

Double-Take Availability overview

Double-Take Availability is a real-time data replication and failover software product. It augments your existing data protection strategy by reducing downtime and data loss, and it provides these services with minimal impact on existing network and communication resources.

Double-Take Availability allows you to identify and protect workloads by replicating, in real-time, data from a production server, known as the source, to a backup server, known as the target. The target server, on a local network or at a remote site, stores the copy of the data from the source. Double-Take Availability monitors any changes to the data and sends the changes to the target server. By replicating only the file changes rather than copying an entire file, Double-Take Availability allows you to more efficiently use resources.

Core operations

Double-Take Availability performs four basic types of operations.

Mirroring

Mirroring is the process of transmitting user-specified data from the source to the target so that an identical copy of data exists on the target. When Double-Take Availability initially performs mirroring, it copies all of the selected data, including file attributes and permissions. Mirroring creates a foundation upon which Double-Take Availability can efficiently update the target server by replicating only file changes.

If subsequent mirroring operations are necessary, Double-Take Availability can mirror specific files or blocks of changed data within files. By mirroring only files that have changed, network administrators can expedite the mirroring of data on the source and target servers.

Mirroring has a defined end point - when all of the selected files from the source have been transmitted to the target. When a mirror is complete, the target contains a copy of the source files at that point in time.

Replication

Replication is the real-time transmission of file changes. Unlike other related technologies, which are based on a disk driver or a specific application, the Double-Take Availability replication process operates at the file system level and is able to track file changes independently from the file’s related application. In terms of network resources and time, replicating changes is a more efficient method of maintaining a real-time copy of data than copying an entire file that has changed.

After a source and target have been connected through Double-Take Availability, file system changes from the user-defined data set are tracked. Double-Take Availability immediately transmits these file changes to the target server. This real-time replication keeps the data on the target up-to-date with the source and provides high availability and disaster recovery with minimal data loss.

Unlike mirroring which is complete when all of the files have been transmitted to the target, replication continuously captures the changes as they are written to the source. Replication keeps the target up-to-date and synchronized with the source.

Failure monitoring and failover

Failover is the process in which a target stands in for a failed source. As a result, user and application requests that are directed to the failed source are routed to the target.

Double-Take Availability monitors the source status by tracking network requests and responses exchanged between the source and target. When a monitored source misses a user-defined number of requests, Double-Take Availability assumes that the server has failed. Double-Take Availability then prompts the network administrator to initiate failover, or, if configured, it occurs automatically.

The failover target assumes the network identity of the failed source. When the target assumes the identity of the source, user and application requests destined for the source server or its IP address(es) are routed to the target.

When partnered with the Double-Take Availability data replication capabilities, failover routes user and application requests with minimal disruption and little or no data loss. In some cases, failover may be used without data replication to ensure high availability on a server that only provides processing services, such as a web server.

Restoration

Restoration provides an easy method for copying replicated data from the target back to its original location on the source. The process only requires you to select the source, target, and the appropriate replication set. There is no need to select files or to remember where the data came from on the source since that information is maintained by Double-Take Availability.

Restoration can be used if the source data is lost due to a disk crash or when the most up-to-date data exists on the target due to failover. At the time of a source server failure, your Double-Take Availability target will contain the same data as your Double-Take Availability source. If you are using the Double-Take Availability failover capabilities, users can continue updating data on the target server while the problems on the source are resolved. Because of the continued updates on the target, when the source server is ready to come back online, the two servers will no longer contain the same data. Restoration is the process of copying the up-to-date data from the target back to the original source or a new source.

When a restoration is complete, the source and target are again synchronized. Replication continues from the target to the source, keeping the two servers synchronized, until you disconnect the restoration connection.

Full-server protection

In addition to protecting your data, you can also protect an entire server. An image of the entire source is replicated to the target. The image contains all of the source server's data and the source server's system state, which is the server's configured operating system and the installed applications. As changes are made on the source, Double-Take Availability mirroring and replication keeps the image of the source stored on the target up-to-date.

 

 

Configurations

Double-Take Availability is an exceptionally flexible product that can be used in a wide variety of network configurations. To implement Double-Take Availability effectively, it is important to understand the possible configuration options and their relative benefits. Double-Take Availability configuration options can be used independently or in varying combinations.

One-to-one, active/standby

Description One target server, having no production activity, is dedicated to support one source server. The source is the only server actively replicating data.
Applications

This configuration is appropriate for offsite disaster recovery, failover, and critical data backup. This is especially appropriate for critical application servers such as Exchange, SQL Server, and web servers.

This is the easiest configuration to implement, support, and maintain.

Considerations This configuration requires the highest hardware cost because a target server is required for every source server.
One-to-one, active/active

Description Each server acts as both a source and target actively replicating data to each other
Applications This configuration is appropriate for failover and critical data backup. This configuration is more cost-effective than the Active/Standby configuration because there is no need to buy a dedicated target server for each source. In this case, both servers can do full-time production work.
Considerations

Coordination of the configuration of Double-Take Availability and other applications can be more complex than the one-to-one active/standby configuration.

During replication, each server must continue to process its normal workload.

Administrators must avoid selecting a target destination path that is included in the source’s replication set. Any overlap will cause an infinite loop.

To support the production activities of both servers during failover without reducing performance, each server should have sufficient disk space and processing resources.

Failover and failback scripts must be implemented to avoid conflict with the existing production applications.

Many-to-one

Description Many source servers are protected by one target server.
Applications This configuration is appropriate for offsite disaster recovery. This is also an excellent choice for providing centralized tape backup because it spreads the cost of one target server among many source servers.
Considerations

The target server must be carefully managed. It must have enough disk space and RAM to support replication from all of the source systems. The target must be able to accommodate traffic from all of the servers simultaneously.

If using failover, scripts must be coordinated to ensure that, in the event that the target server stands in for a failed server, applications will not conflict.

One-to-many

Description One source server sends data to multiple target servers. The target servers may or may not be accessible by one another.
Applications This configuration provides offsite disaster recovery, redundant backups, and data distribution. For example, this configuration can replicate all data to a local target server and separately replicate a subset of the mission-critical data to an offsite disaster recovery server.
Considerations Updates are transmitted multiple times across the network. If one of the target servers is on a WAN, the source server is burdened with WAN communications.
Chained

Description The source servers sends replicated data to a target server, which acts as a source server and sends data to a final target server, which is often offsite.
Applications This is a convenient approach for integrating local high availability with offsite disaster recovery. This configuration moves the processing burden of WAN communications from the source server to the target/source server. After failover in a one-to-one, many-to-one, or one-to-many configuration, the data on the target is no longer protected. This configuration allows failover from the first source to the middle machine, with the third machine still protecting the data.
Considerations The target/source server could become a single point of failure for offsite data protection.

Double-Take Availability requirements

Each Double-Take Availability server must meet minimum requirements. Verify that each server meets the requirements for the function of that machine. Additionally, the machine where you will be running the console must also meet some basic requirements.

Source and target server requirements Console requirements

The Double-Take Availability Replication Console can be run on any of the following operating systems.

Documentation requirements

In order to view the online documentation, you must have one of following browsers.

Installation

Review Double-Take Availability requirements and Installation and upgrade notes then use the appropriate instructions from the following list to meet your goal.

Installation and upgrade notes

Review the following installation and upgrade notes before beginning your installation or upgrade.

Installing or upgrading Double-Take Availability Linux servers

Use these instructions if you are installing or upgrading Double-Take Availability on a Linux server.

  1. Determine the installation package that is appropriate for your operating system.
Operating System Architecture Installation File
32-bit operating system

DoubleTake-version_numbers.i386.rpm

64-bit operating system DoubleTake-version_numbers.x86_64.rpm

The version_numbers in the installation file name will vary and will correspond to the version of Double-Take Availability you are installing. For example, if you are installing version 4.6.11.0.2200, the installation files would be DoubleTake-4.6.11.0-2200.i386.rpm or DoubleTake-4.6.11.0-2200.x86_64.rpm. If you are uncertain about the architecture of your machine, you can use the uname -m command to determine it.

  1. If you are upgrading an existing Double-Take Availability installation, you should complete the following steps before upgrading.
  1. Once you have determined the appropriate installation package to use, you can install the software from the UI or from the command line.
  1. Go to a shell prompt by launching a terminal application from your UI or logging in via the Linux virtual console.
  2. If you are not already, become the superuser ("root") by typing su -.
  3. If you are installing from a CD, mount the file ISO 9660 or UDF file system.
  4. To run the installation use rpm -i with the installation file name to install the software or rpm -U with the installation file name to upgrade the software. For example, if you were installing on a 32-bit operating system, you would use the command rpm -i DoubleTake-4.6.11.0-2200.i386.rpm.

A successful installation returns you to the shell prompt. If you receive an error message during the installation, you will need to reinstall the software. If you are unable to resolve the error, contact technical support.

Use DTSetup to configure your servers for Double-Take Availability.

Installing or upgrading Double-Take Availability Windows client

Use the instructions to install or upgrade the Double-Take Availability client on a Microsoft Windows machine.

  1. Close any open applications.
  2. Start the installation program using the appropriate instructions, depending on your media source.
  1. When the Double-Take Availability installation program begins, the Welcome screen is displayed. Click Nextto continue.
  2. Review and accept the Double-Take Software license agreement to continue with the installation program. Click Next to continue.
  3. Select the folder where you would like to install the Double-Take Availability clients and click Next to continue.
  4. When you are ready to begin copying the files, click Install.
  5. After the files have completed copying, click Finish to exit the installation program.

Removing (erasing) Double-Take Availability

Use these instructions if you want to remove (erase) an existing Double-Take Availability installation.

  1. Unmount DTFS.
  2. Go to a shell prompt by launching a terminal application from your graphical user interface (GUI) or logging in via the Linux virtual console.
  3. If you are not already, become the superuser ("root") by typing su -.
  4. Erase Double-Take Availability by using the command rpm - e DoubleTake.

A successful removal returns you to the shell prompt. If you receive an error message during the removal, you will need to erase the software again. If you are unable to resolve the error, contact technical support.

Note: After removing Double-Take Availability, some files may not be removed until after the server has been rebooted

 

DTSetup

DTSetup is a menu-driven application that provides easy access to Double-Take Availability server configuration. Select a link for more information on DTSetup configuration tasks.

Running DTSetup

  1. Run the DTSetup command from the shell prompt to start DTSetup. The command is case-sensitive.
  2. Note: Do not run DTSetup using the sudo command. Use a real root shell to launch DTSetup instead, either by logging in as root on the console or by using the login session of a non-privileged user to run su - to start a root shell.
  3. The first time you run DTSetup after an installation or upgrade, you will be prompted to review the Double-Take Software license agreement. Review the agreement and accept the terms of agreement by typing yes. You cannot use Double-Take Availability without agreeing to the licensing terms.
  4. When the DTSetup menu appears, enter the number of the menu option you want to access.

Setup tasks

The setup tasks are generally configured once. Some of the configuration options are per server, but the replication configuration options are for those servers in a source role. Select a link below to learn more about that setup task.

 

Activating your server

Before you can use Double-Take Availability, each source and target server must have a valid activation code, which is an alpha-numeric codes that applies the appropriate Double-Take Availability license to your installation.

Note: Server activation can also be completed through the Replication Console. See Licensing a server.
  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Set Activation Code Menu.
  4. Select Set Activation Code in /etc/DT/DT.conf.
  5. Enter your activation code and press Enter. The activation code will automatically be inserted into the configuration file. You are prompted to start the Double-Take service after the first installation, and you must restart the service each time the activation code is modified, such as after an upgrade.
  6. Press Enter to return to the menu.
  7. Press Q as many times as needed to return back to the main menu or to exit DTSetup.

Modifying security groups

During the installation, the user root is automatically added to the Double-Take Availability administrators security group. If you want to add other users or remove root, you will need to modify the security group configuration for each source and target server. See Security for more details on the security groups and the privileges granted to each group.

  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Add/Remove users to Double-Take groups.
  4. Select the necessary menu options to add or remove groups to the administrator or monitors group as needed, and specify the user name when prompted.
  5. When you have completed your security group modifications, press Q as many times as needed to return back to the main menu or to exit DTSetup.

Replication configuration

There are three types of replication configurations for your source server.

You cannot establish replication protection for the same set of data using more than one type of replication configuration. For example, /mydata cannot be protected by both file system replication and block device replication. However, you can protect two different file sets using two different replication configurations. For example /mydata1 and /mydata2 could be protected by file system replication and block device replication, respectively.

Configuring file system replication

In order to use Double-Take Availability replication, data that will be replicated must reside on a partition, which is specially mounted with the Double-Take Availability file system (DTFS) driver. DTFS is a transparent file system that monitors the storage file system so that data changes can be captured. Data on the file system can only be replicated if it is under DTFS mount points.

DTSetup allows you to configure entries in /etc/DT/dtfs_mounts to mount specific partitions as DTFS when Double-Take Availability is started.

Note:

When making replication configuration changes, stop any applications that may be running and restart them after the replication changes have been made. Double-Take Availability needs to be loaded on the file system before any applications, otherwise some data may not be replicated.

When protecting an NFS or Samba server, Double-Take Availability should be mounted just above the local file system where the exported NFS or Samba data resides. Additionally, NFS and/or Samba must be started after the Double-Take daemon.

  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Configure File System or Block Device Replication.
  4. Select Configure File System Replication setup file.
  5. To add a DTFS mount, select Add an entry to /etc/DT/dtfs_mounts.
  6. Specify the path of the highest subdirectory for which replication should be captured. This can be the path to a mount point or a subdirectory of a volume.
  7. Note:

    Do not select the root as a replication mount point.

    If you have a single drive system, mount DTFS to any of the subdirectories directly under root.

  8. You will be prompted whether you want to replicate access times. The access-time file property changes during a mirror and causes replication operations to be generated during a mirror and every time a file is read from or written to. This may have a negative effect on performance. If you do not replicate access times on both the source and target when the volume is initially mounted, you may increase the performance, especially during mirroring. (Access times from files closes are always replicated regardless of this configuration.) Specify yes or no to replicate access times.
  9. To remove any DTFS mounts, select Remove /Umount entries in /etc/DT/dtfs_mounts and specify the path that you want to remove.
  10. To immediately mount all of the entries in /etc/DT/dtfs_mounts, select Mount all entries in /etc/DT/dtfs_mounts as DTFS. If you do not select this option, you must stop and restart the daemon for the changes to take effect.
  11. To immediately unmount all of the entries in /etc/DT/dtfs_mounts, select Unmount DTFS from all entries in /etc/DT/dtfs_mounts. This allows you to make changes to DTFS mounts without unloading the daemon.
  12. Note: If you are using Konqueror, you can only unmount a replication mount point by rebooting the server.
  13. When you have completed your file system replication configuration, press Q as many times as needed to return back to the main menu or to exit DTSetup.
Note: You can also mount and unmount DTFS manually using the Setup Tasks, Configure File System or Block Device Replication, Manual Replication Configuration menu option. Changes made from this menu are not persisted between reboots/restarts.

Configuring block device replication

In order to use Double-Take Availability replication, data that will be replicated on a block device must be accessed through a loop device, which is specially attached using the Double-Take Availability loop driver (DTLOOP). DTLOOP allows the loop device to serve as an access point for operations performed on the block device so that data changes can be captured. Existing block devices may be available for replication, but the data on those block devices can only be replicated if they are accessed through the DTLOOP loop device. It is important that operations on the block device be made through the loop device only, or the operations will not be replicated. Failure to do so will result in corrupted data on the target.

DTSetup allows you to configure entries in /etc/DT/dtloop_devices to attach block devices as DTLOOP when Double-Take Availability is started.

Note:

If your block device being protected with DTLOOP has a file system on it, do not mount them from /etc/fstab. They should be mounted from an init script. DTMount must be started in the boot sequence before the script to mount the loop devices is executed in order to ensure that the loop devices have the replicated block devices associated with them. The script should then mount the loop device, not on the native block device.

When making replication configuration changes, stop any applications that may be running and restart them after the replication changes have been made. Double-Take Availability needs to be loaded on the file system before any applications, otherwise some data may not be replicated.

After the block device replication configuration is complete, applications must read and write through the /dev/loop# device in order for replication to work.

  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Configure File System or Block Device Replication.
  4. Select Configure Block Device Replication setup file.
  5. If you want to see a list of block devices to which DTLOOP can be attached, select List block devices on this system. Swap and LVM physical partitions will not be included in the list.
  6. Press Enter to continue.
  7. To add a DTLOOP device, select Add an entry to /etc/DT/dtloop_devices.
  8. Enter the path to the block device that is to be replicated and press Enter.
  9. Enter the path to the loop device to use (/dev/loop#), if the same one should always be attached. DTLOOP can also use the first one available, but that may mean it attaches to a different one on subsequent reboots/restarts, which may not be desirable. Press Enter to continue.
  10. You will be asked if you want to attach at an offset into the block device and if you want to use an encrypted loop device. In general, these options can be left blank. See the losetup man page for more information on using encryption.
  11. To remove any DTLOOP devices, select Remove /Detach entries in /etc/DT/dtloop_devices and specify the path that you want to remove.
  12. To immediately attach all of the entries in /etc/DT/dtloop_devices, select Attach all entries in /etc/DT/dtloop_devices to a loop device. If you do not select this option, you must stop and restart the daemon for the changes to take effect.
  13. To immediately detach all of the entries in /etc/DT/dtloop_devices, select Detach loop devices from all entries in /etc/DT/dtloop_devices. This allows you to make changes to DTLOOP without unloading the daemon.
  14. When you have completed your block device replication configuration, press Q as many times as needed to return back to the main menu or to exit DTSetup.
Note:

You can also attach and detach DTLOOP manually using the Setup Tasks, Configure File System or Block Device Replication, Manual Replication Configuration menu option. Changes made from this menu are not persisted between reboots/restarts.

Configuring full-server protection

In order to use full-sever protection, Double-Take Availabilitymust configure the root file system for replication. In order for the replication changes to be applied, you will have to reboot your server. Ideally, you will want to reboot the server immediately. However, if you have to schedule the reboot for a later time, mirroring will start immediately. Replication will not start until the server is rebooted. Without replication, data integrity cannot be maintained on the target.

Note:

When making replication configuration changes, stop any applications that may be running and restart them after the replication changes have been made. Double-Take Availability needs to be loaded on the file system before any applications, otherwise some data may not be replicated.

Full-server protection can also be completed through the Connection Wizard.

  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Configure File System or Block Device Replication.
  4. Select Full Server Replication Configuration menu.
  5. Select Add Full Server Protection.
  6. You will be prompted if you want to create the replication set. It can be created now, or later through the Connection Wizard. Select yes or not.
  7. If yes chose to create the replication set now, enter a user that is a member of the Double-Take Availability administrators security group, and then specify the password associated with that user. Press Enter to continue. Remember, replication will not be started until the server is rebooted.
  8. If you want to view the contents of the replication set, select View replication set contents. After you have reviewed the contents, press Enter to continue.
  9. If you need to remove full-server protection, select Remove Full Server Protection. Confirm that you want to remove full-server protection, and then press Enter to continue. A server reboot is required to fully remove full-server protection.
  10. If you want to modify any of the data protected manually, you can select Configure /etc/DT/dtfs_mounts manually which will take you to the Configure File System Replication setup file menu.
  11. When you have completed your full-server protection, press Q as many times as needed to return back to the main menu or to exit DTSetup.

Configuring server settings

Optional server settings are available in various places. You can access them via the Replication Console, through DTCL See the Scripting Guide for details on accessing the server settings through DTCL. , or through DTSetup. You may want to modify server settings on both the source and target.

  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Edit Double-Take config file.
  4. The server settings are listed in alphabetical order. Make modifications as necessary, using the control keys specified at the bottom of the page. For a complete list of each server setting, valid values, default values, and optional notes, see Server Settings in the Scripting Guide.
  5. Press control-X to exit the configuration file.
  6. Enter Yes or No to save any changes.
  7. Press Q as many times as needed to return back to the main menu or to exit DTSetup.

Configuring driver performance settings

Driver settings provide configuration flexibility so you can adjust Double-Take Availability based on your servers, network, and replication requirements. You may want to modify driver settings on both the source and target.

Note: Changing the driver performance settings can have a positive or negative impact on server performance. These settings are for advanced users. If you are uncertain how to best modify the driver performance settings, contact technical support.
  1. Start DTSetup.
  2. Select Setup tasks.
  3. Select Configure Double-Take driver performance.
  4. The current driver settings are displayed.
  5. Select a driver setting to modify the option.
  6. Driver Setting Description
    Toggle Adaptive Throttling You can toggle between enabling (true) and disabling (false) Adaptive Throttling. This occurs when kernel memory usage exceeds the Throttling Start Level percentage. When throttling is enabled, operations are delayed by, at most, the amount of time set in Maximum Throttling Delay, thus reducing kernel memory usage. Throttling stops when the kernel memory usage drops below the Throttling Stop Level percentage.
    Toggle Forced Adaptive Throttling You can toggle between enabling (true) and disabling (false) Forced Adaptive Throttling. This causes all operations to be delayed by, at most, the amount of time in set in Maximum Throttling Delay, regardless of the kernel memory being used. Adaptive Throttling must be enabled (true) in order for Forced Adaptive Throttling to work.
    Set Maximum Throttling Delay This option is the maximum time delay, in milliseconds, used by the driver during a system delay.
    Set Throttling Delay Interval

    This option is the interval, in milliseconds, to check memory usage during a throttling delay. If a delay is no longer needed, the remainder of the delay is skipped.

    Set Throttling Start Level Throttling starts when disk writes reach the specified percentage. This prevents the driver from stopping replication because memory has been exhausted.
    Set Throttling Stop Level Throttling stops when disk writes reach the specified percentage.
    Set Memory Usage Limit

    This option is the amount of kernel memory, in bytes, used for queuing replication operations. When this limit is exceeded, the driver will send an error to the daemon forcing a remirror of all active connections.

    Set Maximum Write Buffer Size

    This option is the maximum amount of system memory, in bytes, allowed for a single write operation. Operations exceeding this amount are split into separate operations in the queue.

  7. After you have completed your driver performance modifications, press Q as many times as needed to return back to the main menu or to exit DTSetup.

Starting and stopping the daemon

The Double-Take daemon will start automatically after Double-Take Availability is installed and the server is rebooted. You can start and stop the Double-Take daemon using this built-in DTSetup script.

  1. Start DTSetup.
  2. Select Starget/Stop Double-Take daemon.
  3. Select the necessary menu option to start or stop the daemon and handle the driver configuration.
  1. When you have completed your starting and stopping tasks, press Q as many times as needed to return back to the main menu or to exit DTSetup.

Starting DTCL

You can launch the Double-Take Availability interactive command prompt which allows you to enter DTCL commands one at a time.

  1. Start DTSetup.
  2. Select Start User Interface (DTCL -i).
  3. Enter your DTCL commands one at a time at the Command prompt. For a complete list of DTCL commands, their syntax, and instructions for completing tasks using DTCL, see the Scripting Guide.
  4. To exit the DTCL Command prompt, type exit.
  5. When you have completed your DTCL tasks, press Q as many times as needed to return back to the main menu or to exit DTSetup.

Viewing documentation and troubleshooting tools

  1. Start DTSetup.
  2. Select Documentation/Troubleshooting tasks.
  3. Select View log files to view the following log files. Double-Take Availability logs alerts, which are processing notifications, warnings, and error messages. The logs are written to disk.
  1. Select Collect and package diagnostic info to collect configuration data when reporting problems to technical support. DTSetup can gather Double-Take Availability log files, Double-Take Availability settings, network configuration information such as IP addresses, and other data which may be necessary for technical support to troubleshoot issues.
  2. Select View user documentation to view Double-Take Availability product documentation and several legal documents. DTSetup attempts to determine your viewers, although you can specify your viewer.
  1. When you have completed your documentation and troubleshooting tasks, press Q as many times as needed to return back to the main menu or to exit DTSetup.

DTSetup menus

The following lists is an overview of the DTSetup menu system. Reference the links for complete details on completing tasks in DTSetup.

Setup tasks—Activation codes, security groups, replication configuration, server configuration, and driver performance settings. See Setup tasks.

Set Activation Code Menu—See Activating your server.

Add/Remove users to Double-Take groups—See Adding users to security groups.

Configure File System or Block Device Replication—There are three types of replication configurations for your source server. See Replication configuration.

Configure File System Replication setup file—See Configuring file system replication.

Configure Block Device Replication setup file—See Configuring block device replication.

Manual Replication Configuration menu—Subset of file system and block device configuration, but the changes are not persisted between reboots/restarts. See Configuring file system replication and Configuring block device replication.

Full Server Replication Configuration menu—See Configuring full-server replication.

Edit Double-Take config file—See Configuring server settings.

Configure Double-Take driver performance—See Configuring driver performance settings.

Start/Stop Double-Take daemon—See Starting and stopping the daemon.

Start User Interface (DTCL -i)—See Starting DTCL.

Documentation/Troubleshooting tasks—See Viewing documentation and troubleshooting tools.

Clients

Double-Take Availability has two clients, the Replication Console and the Failover Control Center that control and manage your connections and failover. Both clients can be started from the Windows Start menu. You can also launch the Failover Control Center from the Tools menu in the Replication Console.

Note:

Double-Take Availability also has a scripting language which can be used in an interactive client or in scripts. For more information, see the Scripting Guide.

Replication Console

From the Replication Console, you can manage, monitor, and control your Double-Take Availability connections. The Replication Console is a two pane view. The views in the panes change depending on what is highlighted. For example, when the root of the tree in the left pane is selected, all of the machines in your environment running Double-Take Availability are displayed in the right pane. If you expand the tree in the left pane and select a server, any connections for that server are displayed in the right pane.

Using Replication Console workspaces

The Replication Console workspace contains the display of the panes of the Replication Console and any servers that may have been inserted. Multiple workspaces can be used to help organize your environment or to view settings from another machine.

Clearing stored security credentials

Use the following steps to remove credentials cached in the Replication Console.

  1. To access the credentials security option, select File, Options and select the Security tab.

  1. To remove the security credentials, click Clear Cached Security Credentials.
  2. Click OK.

Failover Control Center

From the Failover Control Center, you can manage, monitor, and control failover for your Double-Take Availability servers. The Failover Control Center displays a main window for monitoring failover activity. Control buttons to the right allow you to configure and manage your servers.

Setting the frequency of Failover Control Center console refreshes

The failover client periodically requests information from the source and target. Depending on the type of information, the request may be a machine-specific request, like obtaining the Time to Fail status from a target, or may be a general request, like determining which machines are running Double-Take Availability.

The rate at which these requests are made can be modified through the Failover Control Center refresh rate dialog box. Select Settings, Refresh Rate. The default update interval is one second. A lower refresh rate value updates the information in the Failover Control Center window's Monitored Machines tree more often, but also generates more network traffic and higher utilization on the client and target machines. A higher refresh rate value updates the information less frequently, but minimizes the network traffic.

Data protection

Protecting your data consists of two main tasks - creating a replication set (to identify the data to protect) and connecting that replication set to a target.

You have the following data protection options.

 

 

Establishing a data connection using the automated Connection Wizard

The Connection Wizard guides you through the process of protecting your data. It helps you select a source, identify the data from your source that will be included in the replication set, and select a target.

  1. Start the Connection Wizard to establish your connection by selecting Tools, Connection Wizard.
  2. Note: If the Servers root is highlighted in the left pane of the Replication Console, the Connection Wizard menu option will not be available. To access the menu, expand the server tree in the left pane, and highlight a server in the tree.
  1. The Connection Wizard opens to the Welcome screen. Review this screen and click Next to continue.
  2. Note: At any time while using the Connection Wizard, click Back to return to previous screens and review your selections.
  1. If you highlighted a source in the Replication Console, the source will already be selected. If it is not, select the Double-Take Availability source. This is the server that you want to protect.
  2. Note:

    Double-Take Availability will automatically attempt to log on to the selected source using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. Click Next to continue.
  2. If you highlighted a target in the Replication Console, the target will already be selected. If it is not, select the Double-Take Availability target. This is your backup server that will protect the source.
  3. Note:

    Double-Take Availability will automatically attempt to log on to the selected target using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. Click Next to continue.
  2. Select the type of connection you want to establish.

Select Protect data and click Next to continue.

  1. Choose to create a new replication set or use a replication set that already exists.
  1. Click Next to continue.
  2. If you are creating a new replication set, a tree display appears identifying the volumes and directories available on your selected source server. Mark the check box of the volumes and/or directories you want to protect and click Next to continue.
  3. Select the location on the target where the data will be stored.
  1. Click Next to continue.
  2. Review your selections on the summary screen. If your Connection Wizard settings are correct, establish your connection by completing one of the two options below:

Creating a replication set

Before you can establish a connection, you must create a replication set.

  1. Highlight a source in the left pane of the Replication Console and select Insert, Replication Set from the menu bar. You can also right-click on the source name and select New, Replication Set.
  2. A replication set icon appears in the left pane under the source. By default, it is named New Replication Set. Rename the newly inserted replication set with a unique name by typing over the default name and pressing Enter. This process is similar to naming a new folder in Windows Explorer.
  3. Expand the tree under the replication set name to view the volume and directory tree for the source.
  4. Note:

    The default number of files that are listed in the right pane of the Replication Console is 2500, but this is user configurable. A larger number of file listings allows you to see more files in the Replication Console, but results in a slower display rate. A smaller number of file listings displays faster, but may not show all files contained in the directory. To change the number of files displayed, select File, Options and adjust the File Listings slider bar to the desired number.

    To hide offline files, such as those generated by snapshot applications, select File, Options and disable Display Offline Files. Offline files and folders are denoted by the arrow over the lower left corner of the folder or file icon.

  1. Identify the data on the source that you want to protect by selecting volumes, drives, directories, and/or specific files.

Note: Be sure and verify what files can be included by reviewing Replication capabilities.
  1. After selecting the data for this replication set, right-click the new replication set icon and select Save. A saved replication set icon will change from red to black.
  2. If you need to select a block device for replication, right-click the replication set and select Add Device.
  3. The block devices configured for Double-Take Availability replication are shown by default. Highlight the device to include in the replication set and click OK.
  4. Note: If the device you want to include is not displayed, you can click Show Other Devices to view all devices which are eligible for Double-Take Availability replication. You can select any of these devices, but you cannot use them for Double-Take Availability replication until they are configured for Double-Take Availability replication. The status no dtloop indicates the device is not configured for Double-Take Availability replication.
  5. Repeat steps 6 and 7 for any additional devices.
  6. Right-click the updated replication set icon and select Save.

Establishing a connection manually using the Connection Manager

After you have created a replication set, you can establish a connection through the Connection Manager by connecting the replication set to a target.

  1. Open the Connection Manager to establish the connection.
  1. The Connection Manager opens to the Servers tab. Depending on how you opened the Connection Manager, some entries on the Servers tab will be completed already. For example, if you accessed the Connection Manager by right-clicking on a replication set, the name of the replication set will be displayed in the Connection Manager. Verify or complete the fields on the Servers tab.

  1. If desired, you can configure mirror settings before establishing your connection. Select the Mirroring tab on the Connection Manager.

  1. Click Connect to establish the connection.

Establishing a connection across a NAT or firewall

If your source and target are on opposite sides of a NAT or firewall, you will need special configurations to accommodate the complex network environment. Additionally, you must have the hardware already in place and know how to configure the hardware ports. If you do not, see the reference manual for your hardware.

In this environment, you must have static mapping where a single, internal IP address is always mapped in a one-to-one correlation to a single, external IP address. Double-Take Availability cannot handle dynamic mappings where a single, internal IP address can be mapped to any one of a group of external IP addresses managed by the router.

  1. Double-Take Availability uses specific ports for communication between the Double-Take Availability servers and Double-Take Availability clients. In order to use Double-Take Availability through a NAT or firewall, you must first verify the current Double-Take Availability port settings so that you can open the correct ports on your hardware to allow Double-Take Availability machines to communicate with each other. Using the following table, locate and record your port settings for each of the Double-Take Availability ports. The port setting can be found in the following locations.
  1. You need to configure your hardware so that Double-Take Availability traffic is permitted access through the router and directed appropriately. Using the port information from the previous section, configure your router identifying each Double-Take Availability server, its IP address, and the Double-Take Availability and router ports. Also, note the following caveats.

Since there are many types of hardware on the market, each can be configured differently. See your hardware reference manual for instructions on setting up your particular router.

  1. If your network is configured to propagate UDP broadcasts, your servers will be populated in the Replication Console from across the router. If not, you have to manually insert the servers, by selecting Insert, Server. Type the IP address of the router the server is connected to and the port number the server is using for heartbeats.
  2. Once your server is inserted in the Replication Console, you can use the Connection Wizard or the Connection Manager to establish your connection.

Simulating a connection

Double-Take Availability offers a simple way for you to simulate a connection in order to generate statistics that can be used to approximate the time and amount of bandwidth that the connection will use when actively established. This connection uses the TDU (Throughput Diagnostics Utility), which is a built-in null (non-existent) target to simulate a real connection. No data is actually transmitted across the network. Since there is no true connection, this connection type helps you plan for a disaster recovery solution.

  1. Before and after simulating your connection, you should gather network and system information specific to Double-Take Availability operations. Use the DTSetup Diagnostics utility to automatically collect this data. It gathers Double-Take Availability log files, Double-Take Availability and system settings, network configuration information, and other data which may be necessary in evaluating Double-Take Availability performance. Run the DTSetup command from the shell prompt.
  2. Note: Do not run DTSetup using the sudo command. Use a real root shell to launch DTSetup instead, either by logging in as root on the console or by using the login session of a non-privileged user to run su - to start a root shell.
  3. Select DTSetup option 4 for troubleshooting, then select option 2 for diagnostics.
  4. When you run the diagnostics, it may take several minutes for it to finish processing. When it is complete, a .tar.gz file will be created in /var/run/etc/DT/. The file name will have DTInfo with the date and time.
  5. Opening the Connection Manager to establish the connection.
  1. The Connection Manager opens to the Servers tab. Depending on how you opened the Connection Manager, some entries on the Servers tab will be completed already. For example, if you accessed the Connection Manager by right-clicking on a replication set, the name of the replication set will be displayed in the Connection Manager. Verify or complete the fields on the Servers tab.
  1. Click Connect to establish the connection. The simulation data will be logged to the Double-Take Availability statistics file.
  2. Repeat steps 1-3 to run the diagnostics utility after the simulation is complete.

Establishing full-server protection

To establish full-server protection, you will use the Connection Wizard which will guide you through the process of protecting your entire server. It helps you select a source and target, and then it automatically creates the replication set for you.

  1. Start the Connection Wizard to establish your connection by selecting Tools, Connection Wizard.
  2. Note: If the Servers root is highlighted in the left pane of the Replication Console, the Connection Wizard menu option will not be available. To access the menu, expand the server tree in the left pane, and highlight a server in the tree.
  1. The Connection Wizard opens to the Welcome screen. Review this screen and click Next to continue.
  2. Note: At any time while using the Connection Wizard, click Back to return to previous screens and review your selections.
  1. If you highlighted a source in the Replication Console, the source will already be selected. If it is not, select the Double-Take Availability source. This is the server that you want to protect.
  2. Note:

    Double-Take Availability will automatically attempt to log on to the selected source using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. Click Next to continue.
  2. If you highlighted a target in the Replication Console, the target will already be selected. If it is not, select the Double-Take Availability target. This is your backup server that will protect the source.
  3. Note:

    Double-Take Availability will automatically attempt to log on to the selected target using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. Click Next to continue.
  2. Select the type of connection you want to establish.

Select Protect full server including system state and data. Once you proceed, Double-Take Availability will automatically configure the root file system for replication. In order for the replication changes to be applied, you will have to reboot your server. Ideally, you will want to reboot the server immediately. However, if you have to schedule the reboot for a later time, mirroring will start immediately. Replication will not start until the server is rebooted. Without replication, data integrity cannot be maintained on the target.

Note If you need to remove the full server protection configuration, you must use DTSetup.
  1. Click Next to continue.
  2. Reboot your source server now if possible, and then acknowledge the reboot prompt. Remember, you must reboot your source to start replication.
  3. By default, all volumes are automatically selected for protection. You can deselect those volumes that you do not want to protect by clearing the checkbox, although if those volumes contain application data, the integrity of your installed applications will not be maintained. You cannot clear the checkbox for the boot volume because that is where the system state information is stored. A replication set will automatically be created using the volumes you select. Click Next to continue.
  4. Select a location on the target to store the source image and data. In order to distinguish this image from images of other servers, the source name name will automatically be appended to the path you select and will be displayed on-screen. Click Next to continue.
  5. Review your selections on the summary screen. If your Connection Wizard settings are correct, establish your connection by completing one of the two options below:

Protection monitoring

Double-Take Availability flexibility offers a wide variety of methods for monitoring your protection.

 

Monitoring a connection through the Replication Console

When a source is highlighted in the left pane of the Replication Console, the connections and their statistics are displayed in the right pane. Additionally, colors and icons are used for the connections, and the Double-Take Availability servers, to help you monitor your connections.

Connection statistics
  1. You can change the statistics that are displayed by selecting File, Options and selecting the Statistics tab.
  2. The statistics displayed in the Replication Console will be listed with check boxes to the left of each item. Mark the check box to the left of each statistic that you want to appear, and clear the check box to the left of each statistic that you do not want to appear.
  3. The statistics appear on the Replication Console in the order they appear on the Statistics tab. If you want to reorder the statistics, highlight the statistic to be moved and select the up or down arrow button, to the right of the vertical scroll bar, to move the selection up or down in the list. Repeat this process for each statistic that needs to be moved until you reach the desired order.
  4. If you have made changes to the statistics list and have not yet saved them, you can go back to the previously used settings by clicking Reset to Last. This will revert the list back to the last saved settings.
  5. To return the statistics list to the Double-Take Availability default selection and order, click Reset to Default.
  6. Click OK to apply and save any changes that have been made to the order or display of the Replication Console statistics.

Statistics marked with an asterisk (*) are not displayed, by default.

Statistic Description
Replication Set Replication set indicates the name of the connected replication set.
Connection ID The connection ID is the incremental counter used to number each connection established. This number is reset to one each time the Double-Take daemon is restarted.
Target Name The name of the target as it appears in the server tree in the left pane of the Replication Console. If the server’s name is not in the server tree, the IP address will be displayed.
Target IP The target IP is the IP address on the target machine where the mirroring and replication data is being transmitted.
Target Data State
  • OK—The data on the target is in a good state.
  • Mirroring—The target is in the middle of a mirror process. The data will not be in a good state until the mirror is complete.
  • Mirror Required—The data on the target is not in a good state because a remirror is required. This may be caused by an incomplete or stopped mirror or an operation may have been dropped on the target.
  • Restore required—The data on the source and target do not match because of a failover condition. Restore the data from the target back to the source. If you want to discard the changes on the target, you can remirror to resynchronize the source and target.
Target Status
  • OK—The target machine is active and online.
  • Not Loaded—The target module is not loaded on the target. (For example, the activation code is invalid.)
  • Paused—The target machine is paused by user intervention.
  • Retrying—The target machine is retrying operations for the connection.

This field may not be updated until there is source/target activity.

Commit Mode *

The commit mode status indicates the connection status.

  • Real-time—Data is being transmitted to the target machine in real-time.
  • Scheduled—Data is waiting to be transmitted to the target machine until one or more transmit options have been met.
Transmit Mode
  • Started—Data is being transferred to the target machine.
  • Paused—If the transmission is real-time and the transmission has been paused, the Transmit Mode indicates Paused.
  • Scheduled—If the transmission is scheduled, the Transmit Mode indicates Scheduled.
  • Stopped—Data is not being transferred to the target machine.
  • Error—There is a transmission error.
Mirror Status
  • Mirroring—If the file size of the replication set has not been calculated and the data is being mirrored to the target machine, the Mirror Status will indicate Mirroring.
  • Idle—Data is not being mirrored to the target machine.
  • Paused—Mirroring has been paused.
  • Percentage Complete—If the file size of the replication set has been calculated and the data is being mirrored to the target machine, the Mirror Status will display the percentage of the replication set that has been sent.
  • Waiting—Mirroring is complete, but data is still being written to the target.
  • Restoring—Data is being restored from the target to the source.
  • Verifying—Data is being verified.
  • Removing Orphans—Double-Take Availability is checking for orphan files within the target path location (files that exist on the target but not on the source). These files will be removed.
Replication Status
  • Replicating—Data is being replicated to the target machine.
  • Ready—There is no data to replicate to the target machine.
  • Stopped—Replication has stopped.
  • Pending—If auto-remirror is enabled and you have experienced a source or target failure and recovery, the status will change to pending while the connections are reestablished and will update when the remirror begins. If auto-remirror is disabled and you have experienced a source or target failure and recovery, replication will be Pending until a remirror is performed. Without a remirror, data integrity cannot be guaranteed.
  • Out of Memory—Kernel memory has been exhausted.
Queued (Ops) * The queued (ops) statistic indicates the total number of mirror and replication operations that are in the source queue.
Sent (Bytes) The sent (bytes) statistic indicates the total number of mirror and replication bytes that have been transmitted to the target.
Sent Compressed (Bytes) The sent compressed (bytes) statistic indicates the total number of compressed mirror and replication bytes that have been transmitted to the target. If compression is disabled, this statistic will be the same as sent (bytes).
Intermediate Queue (Bytes) * The intermediate queue (bytes) indicates the total amount of memory being used by the operations buffer queue.
Disk Queue (Bytes) The disk queue (bytes) indicates the amount of disk being used to queue data on the source.
Queued Replication (Bytes) The queued replication (bytes) statistic is the total number of replication bytes that are remaining to be transmitted from the source.
Sent Replication (Bytes) The sent replication (bytes) statistic is the total number of replication bytes that have been transmitted to the target.
Sent Compressed Replication (Bytes) * The sent compressed replication (bytes) statistic is the total number of compressed replication bytes that have been transmitted to the target. If compression is disabled, this statistic will be the same as sent replication (bytes).
Queued Mirror (Ops) * The queue mirror (ops) statistic is the total number of mirror operations in the queue.
Sent Mirror (Bytes) The sent mirror (bytes) statistic is the total number of mirror bytes that have been transmitted to the target.
Sent Compressed Mirror (Bytes) * The sent compressed mirror (bytes) statistic is the total number of compressed mirror bytes that have been transmitted to the target. If compression is disabled, this statistic will be the same as sent mirror (bytes).
Skipped Mirror (Bytes) The skipped mirror (bytes) statistic is the total number of bytes that have been skipped when performing a difference or checksum mirror. These bytes are skipped because the data is not different on the source and target machines.
Remaining Mirror (Bytes) The remaining mirror (bytes) statistic is the total number of mirror bytes that are remaining to be sent to the target.
Queued Replication (Ops) * The queued replication (ops) statistic is the total number of replication operations in the queue.
Last File Touched The last file touched identifies the last file that Double-Take Availability transmitted to the target. If you are using long file names (more than several thousand characters long) you may want to disable the display of this statistic to improve Replication Console response times.
Connected Since Connected since is the date and time indicating when the current connection was made. This field is blank, indicating that a TCP/IP socket is not present, when the connection is waiting on transmit options or if the transmission has been stopped. This field will maintain the date and time, indicating that a TCP/IP socket is present, when transmission has been paused.
Connection and sever display

You can configure when the icons and colors change to accommodate your network environment. For example, a slow or busy network may need longer delays before updating the icons or colors.

  1. Select File, Options. On the Configuration tab, you will see Site Monitor and Connection Monitor. The Site Monitor fields control the icons on the left pane of the Replication Console and the icons on the right pane when a group is highlighted in the left pane. The Connection Monitor field controls the display when a server is highlighted in the left pane. These two separate monitoring capabilities allow for flexible monitoring.
  2. Under Site Monitor, specify Check Status Interval to identify the number of seconds between requests sent from the Replication Console to the servers in order to update the display. Valid values are between 0 and 3600. The default setting is 30 seconds.
  3. Under Site Monitor, specify Missed Status Responses to identify the number of responses from a server that can be missed before the Replication Console considers communications lost and updates the icons. Valid values are between 1 and 100. The default setting is 2.
  4. Under Connection Monitor, specify Missed Status Responses to identify the number of responses from a server that can be missed before the Replication Console considers communications lost and updates the icons and colors. Valid values are between 0 and 1000. The default setting is 5.
  5. Click OK to save the settings.
  6. Note:

    If the Site Monitor and Connection Monitor settings are different, at times, the icons and color may not be synchronized between the left and right panes.

Location Icon or Color Description
Icons and colors displayed in the right pane when a server is highlighted in the left pane A green checkmark on a connection indicates the connection is working properly.
A red X on a connection indicates a connection error. For example, an error may be caused by broken transmission or pending replication. To determine the exact problem, locate the connection data item that appears in red.
White If the connection background is white, the Replication Console and the source are communicating.
Gray If the connection background is gray, the Replication Console and the source are no longer communicating. The connection data stops updating once communications have stopped. Once communications have been reestablished, the connection background will change back to white.
Left pane icon An icon with yellow and blue servers indicates a server that is working properly.
A red X on a server icon indicates the Replication Console cannot communicate with that server or that is a problem with one of the server’s connections. If the connection background is gray, it is a communication issue. If the connection also has a red X, it is a connection issue.
A red tree view (folder structure) on a server icon indicates a restore is required because of a failover. You can override the restore required flag by starting a mirror, however, any data changes on the target that may have occurred after failover may be lost.
A black X on a server icon indicates the server is not running Double-Take Availability.

Log files

Various Double-Take Availability components (Double-Take daemon, Replication Console, Failover Control Center, and the Command Line Client) generate a log file to gather alerts, which are notification, warning, and error messages. The log files are written to disk.

Each log file consists of a base name, a series number, and an extension.

Viewing the log files through a text editor

The log files can be viewed, from the location where Double-Take Availability is installed, with a standard text editor.

The following table describes the information found in each column of the log file.

Column Number Description
1 The date the message was generated
2 The time the message was generated.
3 The process ID.
4 The thread ID.
5 The sequence number is an incremental counter that assigns a unique number to each message.
6

The type or level of message displayed.

  • 1—Warning or error message
  • 2—Informational message
7 The message ID.
8 The message text.
Sample Double-Take Availability log file

01/15/2009 14:14:18.3900 95 98 2 2 69 Kernel Started

01/15/2009 14:14:18.4200 95 98 3 2 10004 Valid Activation Key Detected :

01/15/2009 14:14:18.5350 98 170 4 2 52501 Target module loaded successfully

01/15/2009 14:14:18.6760 98 172 5 2 10004 Valid Activation Key Detected :

01/15/2009 14:14:18.9870 130 131 6 2 51501 Source module loaded successfully

01/15/2009 14:24:15.2070 130 132 7 2 72 Connection Request from ip://206.31.4.305

01/15/2009 14:24:16.3090 131 133 8 2 600002 Unified login provides ADMIN access

01/15/2009 14:24:40.9680 132 134 9 2 99 RepSet Modified: UserData

01/15/2009 14:25:22.4070 134 131 10 2 71 Originator Attempting ip://206.31.4.305 01/15/2009 14:25:22.5030 134 131 11 2 0 Transmission Create to ip://206.31.4.305.

01/15/2009 14:25:22.6060 135 133 12 2 500000 UserData is connected to ip://206.31.4.305 01/15/2009 14:25:23.5030 136 98 13 2 87 Start Replication on connection 1

Sample Replication Console log file

00/00/0000 00:00:00.0000 Application starting

09/11/2009 12:45:53.8980 704 1032 1 2 0 Could not find XML file: C:\Program Files\DoubleTake\Administrator.xml, default groups will be added.

09/11/2009 12:45:53.9580 704 1032 2 2 0 Adding default group: Double-Take Servers

09/11/2009 12:45:53.9580 704 1032 3 2 0 Adding default group: Double-Take Servers\

Auto-Discovered Servers

09/11/2009 12:46:08.3390 704 1032 4 210004 Evaluation license expires in 95 day(s).

Viewing the Double-Take Availability log file through the Replication Console

In addition to the statistics and status shown in the Replication Console, you can also open a message window to view the Double-Take Availability log file.

Note:

The standard appearance of the message window is a white background. If your message window has a gray background, the window is inactive. The Replication Console may have lost communications with that server, for example, or you may no longer be logged into that server.

The message window is limited to the most recent 1000 lines. If any data is missing an entry in red will indicate the missing data. Regardless of the state of the message window, all data is maintained in the Double-Take Availability log on the server.

  1. Open a new message window by using any of the following methods:

  1. Repeat step 1 to open multiple message windows.
  2. To control the window after it is created, use either of the following methods to access the control methods listed in the following table.
  1. To change which server you are viewing messages for, select a different machine from the drop down list on the toolbar. If necessary, the login process will be initiated.
  2. To move the message window to other locations on your desktop, click and drag it to another area or double-click it to automatically undock it from the Replication Console.

Configuring the properties of the Double-Take Availability log file

  1. To modify the maximum file size and the number of Double-Take Availability log files that are maintained, access the Server Properties dialog box by right-clicking a machine name in the left pane of the Replication Console and selecting Properties.
  2. Select the Logging tab.
  3. At the top of the window, Folder indicates the directory where the log files are located. The default is the directory where the Double-Take Availability program files are installed.
  4. Modify any of the options under Messages and Alerts, if necessary.
  1. Click OK to save the changes.

Double-Take Availability log messages

The following table describes some of the standard Double-Take Availability alerts that may be displayed in the log files. The ID appears in column 7 of the log file, and the message appears in column 8.

Note: In the following table, con_id refers to the unique connection ID assigned to each connection between a source replication set and a target.
ID Message Description
0 N/A There are several log messages with this ID#. See the description in the Message column in the log file.
7
  • Synchronous ioctl returned STATUS_PENDING
  • Failed to reset Replication Flags. Replication may not be performed correctly.

 

  • Communication with the Double-Take Availability driver is not being performed correctly. A reboot is required to guarantee replication and data integrity.
  • An error occurred between the Double-Take Availability driver and recent changes to the replication set. The possible resolutions are:
  • Undo the changes to the replication set
  • Stop and restart Double-Take Availability
  • Reboot the server
69 Double-Take kernel started on server_name The Double-Take daemon was started on the Double-Take Availability server specified.
70 Double-Take kernel stopped The Double-Take daemon was stopped on a Double-Take Availability server.
71 Originator attempting ip://xxx.xxx.xxx.xxx A source is requesting to connect a replication set to a target machine.
72 Connection request from ip://xxx.xxx.xxx.xxx A target machine has received a source machine’s request to connect a replication set to the target.
73 Connected to ip://xxx.xxx.xxx.xxx A source machine has successfully connected a replication set to a target machine.
74 Connection paused with ip://xxx.xxx.xxx.xxx A network connection between the source and the target exists and is available for data transmission, but data is being held in queue and is not being transmitted to the target. This happens because the target machine cannot write data to disk fast enough. Double-Take Availability will resolve this issue on its own by transmitting the data in queue when the target catches up.
75 Connection resumed with ip://xxx.xxx.xxx.xxx The transmission of data from the source machine to the target machine has resumed.
76 Connection failed to ip://xxx.xxx.xxx.xxx An attempt to establish a network connection between a source machine and target machine has failed. Check your network connections and verify that the target machine is still online.
77 Connection lost with IP address address The network connection previously established between a source machine and target machine has been lost. Check your network connections and troubleshoot to see why the connection was lost.
78 Auto-disconnect threshold has been reached. The Double-Take Availability queue has exceeded its limit, and the auto-disconnect process will disconnect the source and target connection. The auto-reconnect process will automatically reestablish the connection if the auto-reconnect feature is enabled. If the auto-reconnect feature is not enabled, you must first verify that the connection between the source and target has been broken, and then manually reestablish the connection in the Replication Console.
79 Memory freed to bring Double-Take memory usage below the limit Data in the source queue has been sent to the target machine, bringing the pagefile below its limit.
80 Trying to auto-retransmit to ip://xxx.xxx.xxx.xxx Double-Take Availability is attempting to automatically reconnect previously established source and target connections after a server reboot or auto-disconnect. This is also referred to as the auto-reconnect process.
81 Schedule transmit start to target A scheduled transmission of data from a source machine to a target machine has started. See the description in the Message column in the log file.
82 Schedule transmit end to target A scheduled transmission of data from a source machine to a target machine has ended. See the description in the Message column in the log file.
85 repset has been auto-disconnected Double-Take Availability automatically disconnects the source and target connection because the queue size has reached a specified size for this action.
87 Start replication on connection con_id Data has started replicating from a source machine to a target machine.
88 Stop replication on connection con_id Data has stopped replicating from a source machine to a target machine.
89 Mirror started con_id Data is being mirrored from a source machine to a target machine.
90 Mirror stopped con_id The process of mirroring data from a source machine to a target machine has stopped due to user intervention or an auto-disconnect. (This means the mirroring process was not completed.)
91 Mirror paused con_id The process of mirroring data from a source machine to a target machine has paused because the target machine cannot write the data to disk fast enough. Double-Take Availability will resolve this issue on its own by transmitting the data in queue when the target catches up.
92 Mirror resumed con_id The process of mirroring data from a source machine to a target machine has resumed.
93 Mirror ended con_id The process of mirroring data from a source machine to a target machine has ended.
94 Verification started con_id The verification process of confirming that the Double-Take Availability data on the target is identical to the data on the source has started.
95 Verification ended con_id The verification process of confirming that the Double-Take Availability data on the target is identical to the data on the source has ended.
97 Restore started con_id The restoration process of copying the up-to-date data from the target back to the original source machine has started.
98 Restore completed con_id The restoration process of copying the up-to-date data from the target back to the original source machine has been completed.
99 RepSet Modified: repset_ name This message means that the specified replication set has been modified.
100 Failover condition has been met and user intervention is required Double-Take Availability has determined that the source has failed, and requires manual intervention to start the failover process.
101 Failover in progress!!! The conditions for failover to occur have been met, and the failover process has started.
102 Target full! The disk to which data is being written on the target is full. This issue may be resolved by deleting files on the target machine or by adding another disk.
801 Auto-disconnect has occurred on IP address address with connection con_id Disconnected replication set name: repset_name. Auto-disconnect has occurred for the specified connection. This is due to the source queue filling up because of a network or target failure or bottleneck.
10001 Activation key is not valid. An invalid activation code was identified when the Double-Take daemon was started.
10002 Evaluation period has expired. The evaluation license has expired.
10003 Activation code violation with machine machine_name Duplicate single-server activation codes are being used on the servers, and Double-Take Availability is disabled.
10004 Valid activation key detected A valid activation code was identified when the Double-Take daemon was started.
51001 Source module failed to load The Double-Take Availability source module failed to load. Look at previous log messages to determine the reason. (Look for messages that indicate that either the activation code was invalid or the user-configurable source module was not set to load automatically at startup.) The source module may have been configured this way intentionally.
51501 Source module loaded successfully The Double-Take Availability source module was loaded successfully.
51502 Source module already loaded The Double-Take Availability source module was already loaded.
51503 Source module stopped The Double-Take Availability source module stopped.
52000
  • The target has been paused due to manual intervention.
  • The target has been resumed due to manual intervention
The target has been paused or resumed through user intervention.
52000 Unfinished Op error

This error message contains various Microsoft API codes. The text Code -<x> Internal <y> appears at the end of this message. The code value indicates why the operation failed, and the internal value indicates the type of operation that failed. The most common code values that appear in this error message are:

  • (5) Permission denied: The account running the Double-Take daemon does not have permission to update the file specified.
  • (32) Sharing violation: Another application is using a particular file that Double-Take Availability is trying to update. Double-Take Availability will wait and try to update the file later.
  • (112) Disk full: The disk to which data is being written on the target is full. This issue may be resolved by deleting files on the target machine or by adding another disk.
52501 Target module loaded successfully The Double-Take Availability target module was loaded successfully.
52502 Target module already loaded The Double-Take Availability target module was already loaded.
52503 Target module stopped The Double-Take Availability target module stopped.
53001 File was missing from target The verification process confirms that the files on the target are identical to the files on the source. This message would only appear if the verification process showed that a file on the source was missing from the target.
53003 Could not read filename Double-Take Availability could not read a file on the source machine because the file may have been renamed or deleted. For example, temporary files show up in queue but do not show up during transmission. (No user action required.)
54000 Kernel started The Double-Take daemon was started.
54001 Failover module failed to load The Double-Take Availability failover module failed to load. Look at previous log messages to determine the reason.
54503 Failover module stopped The Double-Take Availability failover module stopped.
99001 Starting source module low memory processing The source’s queue is full, and the auto-disconnect process will disconnect the source and target connection. The auto-reconnect process will automatically reestablish the connection if the auto-reconnect feature is enabled. If the auto-reconnect feature is not enabled, you must first verify that the connection between the source and target has been broken, and then manually reestablish the connection in the Replication Console.
99999 Application is terminating normally The Double-Take daemon is shutting down normally.
503010 Asyncloctl for status thread 178 terminated, terminating the status thread A Double-Take Availability process monitors the state of the Double-Take Availability driver. When the daemon is shut down, the driver is shut down, and this process is terminated. (No user action required.)
600002
  • Unified login provides ADMIN access
  • User has level access (x)
  • Using the current login grants ADMIN access.
  • The listed user has listed access level and access level ID.
700000 The source machine source_machine is not responding to a ping. This occurs when all monitored IP addresses on the source machine stop responding to pings. Countdown to failover will begin at the first occurrence and will continue until the source machine responds or until failover occurs.
800000
  • Active Directory GetHostSpns function call failed
  • Active Directory RemoveSpns function call failed
  • Active Directory AddSpns function call failed
  • Double-Take Availability failed to get the host SPN (Service Principal Name) from Active Directory.
  • Double-Take Availability failed to remove an SPN from Active Directory.
  • Double-Take Availability failed to add a host SPN to Active Directory.

Monitoring the Linux system log

An event is a significant occurrence in the system or in an application that requires administrators to be notified. The operating system writes notifications for these events to the Linux system log. The location of the log file depends on the configuration of /etc/syslog.conf, however, by default, it is /var/log/messages. The following table identifies the events generated by Double-Take Availability.

ID Severity Event Message Required Response
1 Error This evaluation period has expired. Mirroring and replication have been stopped. To obtain a license, please contact your vendor. Contact your vendor to purchase either a single or site license.
2 Info. The evaluation period expires in %1 day(s). Contact your vendor before the evaluation period expires to purchase either a single or site license.
3 Info. The evaluation period has been activated and expires in %1 day(s). Contact your vendor before the evaluation period expires to purchase either a single or site license.
4 Warning Duplicate activation codes detected on machine %1 from machine %2. If you have an evaluation license or a site license, no action is necessary. If you have a single license, you must purchase either another single license or a site license.
5 Error This product edition can only be run on Windows 2000 Server or Advanced Server running the Server Appliance Kit. Verify your activation code has been entered correctly and contact technical support.
3000 Info. Logger service was successfully started. No action required.
3001 Info. Logger service was successfully stopped. No action required.
4000 Info. Kernel was successfully started. No action required.
4001 Info. Target service was successfully started. No action required.
4002 Info. Source service was successfully started. No action required.
4003 Info. Source service was successfully stopped. No action required.
4004 Info. Target service was successfully stopped. No action required.
4005 Info. Kernel was successfully stopped. No action required.
4006 Error Service has aborted due to the following unrecoverable error: %1 Restart the Double-Take daemon.
4007 Warning Auto-disconnecting from %1 (%2) for Replication Set %3, ID: %4 due to %5 The connection is auto-disconnecting because the disk-based queue on the source has been filled, the daemon has encountered an unknown file ID, the target server has restarted, or an error has occurred during disk queuing on the source or target (for example, Double-Take Availability cannot read from or write to the transaction log file).
4008 Info. Auto-disconnect has succeeded for %1 (%2) for Replication Set %3, ID: %4 No action required.
4009 Info. Auto-reconnecting Replication Set %1 to %2 (%3) No action required.
4010 Info. Auto-reconnect has succeeded connecting Replication Set %1 to %2 (%3) No action required.
4011 Error Auto-reconnect has failed connecting Replication Set %1 to %2 (%3) Manually reestablish the replication set to target connection.
4014 Info. Service has started network transmission. No action required.
4015 Info. Service has stopped network transmission. No action required.
4016 Info. Service has established a connection to %1 (%2) for Replication Set %3, ID: %4 No action required.
4017 Info. Service has disconnected from %1 (%2) for Replication Set %3, ID: %4 No action required.
4018 Warning %1, however, mirroring and replication have been disabled as a restore is required due to a previous failover. Perform a restoration.
4019 Info. Service has started a mirror to %1 (%2) for Replication Set %3, ID: %4 No action required.
4020 Info. Service has paused a mirror to %1 (%2) for Replication Set %3, ID: %4 No action required.
4021 Info. Service has resumed a mirror to %1 (%2) for Replication Set %3, ID: %4 No action required.
4022 Info. Service has stopped a mirror to %1 for Replication Set %2, ID: %3, %4 No action required.
4023 Info. Service has completed a mirror to %1 %2 for Replication Set %3, ID: %4, %5 No action required.
4024 Info. Service has started Replication to %1 (%2) for Replication Set %3, ID: %4 No action required.
4025 Info. Service has stopped Replication to %1 (%2) for Replication Set %3, ID: %4 No action required.
4026 Info. The target has been paused due to user intervention. No action required.
4027 Info. The target has been resumed due to user intervention. No action required.
4028 Warning Registration of service class with Active Directory failed. Verify that the Active Directory server is up and the service has the proper permissions to update its entries. Verify that the Active Directory server is running and that the Double-Take daemon has permission to update Active Directory.
4029 Warning Registration of service instance with Active Directory failed. Verify that the Active Directory server is up and the service has the proper permissions to update its entries. Verify that the Active Directory server is running and that the Double-Take daemon has permission to update Active Directory.
4030 Error libRSResource.so has an unknown error. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4031 Error libRSResource.so could not be opened. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4032 Error The libRSResource.so component version does not match the component version expected by the product. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4033 Error libRSResource.so build version is invalid. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4034 Error Error verifying the service name. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4035 Error Error verifying the product name. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4036 Error Error verifying the vendor name. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4037 Error Error verifying the vendor URL name. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4038 Error Error verifying the product code. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4039 Error Error while reading libRSResource.so. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4040 Error The product code is illegal for this computer hardware. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4041 Error The product code is illegal for this operating system version. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4042 Error The product code requires installing the Windows Server Appliance Kit. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4043 Error This product can only be run on a limited number of processors and this server exceeds the limit. The product functionality has been disabled. Reinstall the software, using the installation RPMs, to install a new copy of libRSResource.so. Contact technical support for assistance or if this error persists.
4044 Error An error was encountered and replication has been stopped. It is necessary to stop and restart the service to correct this error. Contact technical support if this error persists.
4045 Error %1 value must be between 1025 and 65535. Using default of %2. Verify that the Double-Take Availability port value you are trying to use is within the valid range. If it is not, it will automatically be reset to the default value.
4046 Error This service failed to start because of a possible port conflict. Win32 error: %1 Verify that the Double-Take Availability ports are not conflicting with ports used by other applications.
4047 Error Could not load ZLIB DLL %1. Some levels of compression will not be available. The compression levels available depend on your operating system. You can reinstall the software, using the installation Repair option, to install a new copy of the DynaZip.dll, or contact technical support if this error persists.
4048 Info. Service has started a delete orphans task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4049 Info. Service has paused a delete orphans task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4050 Info. Service has resumed a delete orphans task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4051 Info. Service has stopped a delete orphans task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4052 Info. Service has completed a delete orphans task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4053 Info. Service has started a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4054 Info. Service has paused a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4055 Info. Service has resumed a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4056 Info. Service has stopped a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4057 Info. Service has completed a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4058 Info. Service has started a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4059 Info. Service has paused a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4060 Info. Service has resumed a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4061 Info. Service has stopped a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4062 Info. Service has completed a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required.
4063 Error The product code requires a virtual server environment. The product functionality has been disabled. Check your activation code.
4100 Error Product activation code is invalid. Please check that it is typed correctly and is valid for the version of the operating system in use. Check your activation code.
4101 Error This service will not run on this device. Contact your sales representative for upgrade procedures. Contact technical support.
4110 Warning Target cannot write %1 due to target disk being full. Operation will be retried (%2 times or forever) The disk on the target is full. The operation will be retried according to the TGExecutionRetryLimit setting.
4111 Warning Target can not write %1 due to a sharing violation. Operation will be retried (%2 times or forever) A sharing violation error is prohibiting Double-Take Availability from writing on the target. The operation will be retried according to the TGExecutionRetryLimit setting.
4112 Warning Target can not write %1 due to access denied. Operation will be retried (%2 times or forever) An access denied error is prohibiting Double-Take Availability from writing on the target. The operation will be retried according to the TGExecutionRetryLimit setting.
4113 Warning Target can not write %1 due to an unknown reason. Operation will be retried (%2 times or forever). Please check the log files for further information on the error. An unknown error is prohibiting Double-Take Availability from writing on the target. The operation will be retried according to the TGExecutionRetryLimit setting.
4120 Info. Target write to %1 was completed successfully after %2 retries. No action required.
4150 Error Target write %1 failed after %2 retries and will be discarded. See the event log or log files for error conditions. After correcting the problem, you should re-mirror or run a verify to resynchronize the changes. The operation has been retried according to the TGExecutionRetryLimit setting but was not able to be written to the target and the operation was discarded. Correct the problem and remirror the files.
4200 Info. In band task %1 submitted from %2 by %3 at %4 No action required.
4201 Warning In band task %1 discarded (submitted from %2 by %3 at %4) A task may be discarded in the following scenarios: all connections to a target are manually disconnected, replication is stopped for all connections to a target, or an auto-disconnect occurs. If one of these scenarios did not cause the task to be discarded, contact technical support.
4202 Info. Running %1 in band script: %2 (task %3 submitted from %4 by %5 at %6) No action required.
4203 Info. Completed run of in band script: %1 (exit code %2) No action required.
4204 Error Error running in band script: %1 Review the task and its associated script(s) for syntax errors.
4205 Warning Timeout (%1 seconds) running in band script: %2 The timeout specified for the script to complete has expired. Double-Take Availability will continue with normal processing. You may need to manually terminate the script if it will never complete.
4206 Warning Run timeout disabled for in band script: %1 The timeout period was set to zero (0). Double-Take Availability will not wait for the script to complete before continuing. No action is required.
4207 Warning In band scripts disabled by server - no attempt will be made to run %1 Enable task command processing.
4300 Error A connection request was received on the target before the persistent target paths could be loaded. You may need to disconnect and reconnect your replication set.
4301 Error Unable to block target paths, the driver is unavailable. If you need to block your target paths, contact technical support.
4302 Info. Target Path %1 has been successfully blocked No action required.
4303 Warning Blocking of target path: %1 failed. Error Code: %2 If you need to block your target paths, contact technical support.
4304 Info. Target Path %1 has been successfully unblocked No action required.
4305 Warning Unblocking of target path: %1 failed. Error Code: %2 If you need to unblock your target paths, contact technical support.
4306 Warning Target paths for source %1 (%2) Connection id: %3 are already blocked No action required.
4307 Warning Target paths for source %1 (%2) Connection id: %3 are already unblocked No action required.
4308 Error Error loading target paths for blocking, registry key %1 has been corrupted. If you need to block your target paths, contact technical support.
5000 Info. Server Monitor service was successfully started. No action required.
5001 Info. Server Monitor service was successfully stopped. No action required.
5002 Info. Placeholders were modified to %1. No action required.
5100 Info. Failover completed for %1. No action required.
5101 Info. IP address %1 with subnet mask %2 was added to target machine's %3 adapter. No action required.
5102 Warning %1 has reached a failover condition. A response from the user is required before failover can take place. User intervention has been configured. Open the Failover Control Center and accept or decline the failover prompt.
5103 Info. Started adding drive shares from %1 to %2. No action required.
5104 Info. %1 drive shares were taken over by %2. No action required.
5105 Info. Attempting to run the %1 script. No action required.
5106 Info. The %1 script ran successfully. No action required.
5107 Error Error occurred in running %1 script. Verify that the script identified exists with the proper permissions.
5108 Error The source machine %1 is not responding to a ping. This occurs when all monitored IP addresses on the source machine stop responding to pings. Countdown to failover will begin at the first occurrence and will continue until the source machine responds or until failover occurs.
5109 Error The public NIC on source machine %1 is not responding to a ping. The failover target did not receive an answer to its ping of the source machine. Eventually, a failover will result. Investigate possible errors (down server, network error, etc.).
5200 Info.. Failback completed for %1. No action required.
5201 Info.. IP address %1 was removed from target machine's %2 adapter. No action required.
5202 Error Unable to Failback properly because IP address %1 was missing a corresponding SubNet Mask. Contact technical support.
5300 Info. The following IP address was added to target's monitoring list: %1 No action required.
5301 Info. The following IP address was removed from target's monitoring list: %1 No action required.
5302 Info. Drive share information for %1 has been updated on the target machine. No action required.
5400 Info. Broadcasted new MAC address %1 for IP address %2. No action required.
5500 Warning Could not connect to e-mail server. Check to make sure the SMTP server %1 is available (error code: %2). Double-Take Availability could not connect to your SMTP server or the username and/or password supplied is incorrect. Verify that SMTP server is available and that you have identified it correctly in your e-mail notification configuration. Also verify that your username and password have been entered correctly.
5501 Warning E-mail notification could not be enabled (error code: %1). This alert occurs if there is an unexpected error enabling e-mail notification during daemon startup. Check to see if any other errors related to e-mail notification have been logged. Also, check to make sure the Windows Management Instrumentation (WMI) daemon is enabled. If neither of these apply, contact technical support.
5502 Warning E-mail notification could not be initialized. Check to make sure Internet Explorer 5.0 or later is installed. E-mail notification requires Internet Explorer 5.0 or later. Verify that you have this version or later installed on the Double-Take Availability server.
5503 Warning E-mail notification could not be processed. Check to make sure the correct version of SMTPMail.DLL is registered on the system (error code: %1). If you are using Double-Take Availability 4.4.2.1 or earlier and Windows NT 4.0, e-mail notification requires Windows Management Instrumentation (WMI) to be installed. Verify that you have it installed on the Double-Take Availability server.
5504 Warning Could not load LocalRS.dll (for e-mail notification). This alert occurs if there is an error loading the resource DLL for the daemon. Typically, this is caused by a missing LocalRS.dll file. Reinstall the software, using the installation Repair option, to install a new copy of the LocalRS.dll. Contact technical support if this error persists.
5505 Warning E-mail could not be sent. Check e-mail settings (error code: %1). Verify that the e-mail server that you have identified in your e-mail notification configuration is correct.
5506 Warning One or more required e-mail settings have not been specified (error code: %1). At a minimum, you must specify the e-mail server, the From and To addresses, and at least one type of event to include.
5507 Warning E-mail notification could not be initialized. Check to make sure WMI is installed and available (error code: %1). If you are using Double-Take Availability 4.4.2.1 or earlier and Windows NT 4.0, e-mail notification requires Windows Management Instrumentation (WMI) to be installed. Verify that you have it installed on the Double-Take Availability server.
5508 Warning An error occurred connecting to the WMI namespace. Check to make sure the Windows Management Instrumentation service is not disabled (error code %1). This alert occurs if there is an error with the Windows Management Instrumentation (WMI) service. Verify that you have it installed on the Double-Take Availability server and that it is enabled.
5600 Warning Part or all of the e-mail setting %1 is not in a valid format. Verify that the include categories and exclude ID list are identified and formatted correctly.

E-mailing system messages

You can e-mail system messages to specified addresses. The subject of the e-mail will contain an optional prefix, the server name where the message was logged, the message ID, and the severity level (information, warning, or error). The text of the message will be displayed in the body of the e-mail message.

  1. To enable e-mail notification for a server, right-click the server in the left pane of the Replication Console and select Properties.
  2. Select the E-mail Notification tab.

  1. Select Enable notification.
  2. Note: Any specified notification settings are retained when Enable notification is disabled.
  3. Specify your e-mail settings.

If desired, enter unique text for the Subject Prefix which will be inserted at the front of the subject line for each Double-Take Availability e-mail message. This will help distinguish Double-Take Availability messages from other messages. This field is optional.

If desired, enable Add event description to subject to have the description of the message appended to the end of the subject line. This field is optional.

Statistics

Statistics logging is the process of taking snapshots of Double-Take Availability statistical data. The data can be written to a file for future use. Changes to the statistics file configuration are detected and applied immediately without restarting the Double-Take daemon.

The statistics log file created is a binary file. To view the log file, you must run statistics from the command prompt.

=================================

0/11/09 12:48:05:2040

=================================

SYSTEMALLOCATOR::Total Bytes: 0

IQALLOCATOR::Total Bytes: 0

SECURITY::Logins : 1 FailedLogins : 0

KERNEL::SourceState: 2 TargetState: 1 Start Time: Tue Sep 11 12:45:26 2007

  RepOpsGenerated: 436845 RepBytesGenerated: 0

  MirOpsGenerated: 3316423 MirBytesGenerated: 108352749214952

   FailedMirrorCount: 0 FailedRepCount: 0

   ActFailCount: 0 TargetOpenHandles: 0 DriverQueuePercent: 0

TARGET:: PeerAddress: 10.10.1.104 LocalAddress: 10.10.1.104

   Ops Received: 25 Mirror Ops Received: 23

   Retries: 0 OpsDropped: 0 Ops Remaining: 0

   Orphan Files Removed: 0 Orphan Directories Removed: 0 Orphan Bytes Removed: 0

   Bytes In Target Queue: 0 Bytes In Target Disk Queue: 0

   TasksSucceeded: 0 TasksFailed: 0 TasksIgnored: 0

SOURCE::autoDisConnects : 0 autoReConnects : 1

   lastFileTouched : /var/log/data_file

CONNECTION:: conPeerAddress: 10.10.1.104

   connectTime: Tue Sep 11 12:45:34 2007

   conState: 1 conOpsInCmdQueue: 0 conOpsInAckQueue: 0

   conOpsInRepQueue: 0 conOpsInMirQueue: 0 conBytesInRepQueue: 0

   conOpsTx: 27 conBytesInMirQueue: 0 conBytesTx: 14952687269

   conBytesCompressedTx: 14952

   conOpsRx: 201127 conBytesRx: 647062280 conResentOpCount: 0 conBytesInDiskQueue: 0

   conBandwidthLimit: 429496295 conBytesSkipped: 22867624 conMirrorBytesRemain: 0

   conMirrorPercent: 100.0%

   conTaskCmdsSubmitted: 0 conTaskCmdsQueued: 0

   conTasksSucceeded: 0 conTasksFailed: 0 conTasksIgnored: 0

Configuring the properties of the statistics file

  1. Right-click a machine in the left pane of the Replication Console and select Properties. 
  2. Select the Logging tab.

  1. At the top of the tab, specify the Folder where the log files for messages, alerts, verification, and statistics will be saved.
  2. Under Statistics, specify the following information:
  1. Select the Setup tab.
  2. Verify that Log Statistics Automatically is enabled. If disabled, statistics will not be logged.
  3. Click OK to save the settings.

Viewing the statistics file

The statistics log file created is a binary file. To view the log file, you must run DTStat from a command prompt. From the directory where Double-Take Availability is installed, run the DTStat command.

Command DTSTAT
Description Starts the DTStats statistics logging utility from a command prompt
Syntax DTSTAT [-p][-i <interval>][-n <iterations>][-t <filename>] [-f <filename>] [-s <filename>] [-st <filename>][-IP <address>] [-START <mm/dd/yyyy hh:mm>][-STOP <mm/dd/yyyy hh:mm>] [-SERVER <ip_address> <port_number>]
Options
  • -p—Do not print the output to the screen
  • -i interval—Refresh from shared memory every interval seconds
  • -n iterations—The number of times to repeat the statistics run
  • -t filename—Save the data from memory to the specified binary file filename
  • -f filename—Reads from a previously saved binary file, filename, that was generated using the -t option instead of reading from memory
  • -s filename—Saves only the connection data from the data in memory to an ASCII, comma-delimited file, filename
  • -st filename—Saves only the target data from the data in memory to an ASCII, comma-delimited file, filename
  • -f filename1 -s filename2—Saves only the connection data from a previously saved binary file, filename1, to an ASCII, comma-delimited file, filename2
  • -f filename1 -st filename2—Saves only the target data from a previously saved binary file, filename1, to an ASCII, comma-delimited file, filename2
  • -IP address—Filters out the specified address in the IP address field and prints only those entries. Specify more than one IP address by separating them by a comma.
  • -START mm/dd/yyyy hh:mm—Filters out any data prior to the specified date and time
  • -STOP mm/dd/yyyy hh:mm—Filters out any data after the specified date and time
  • -SERVER ip_address port_number—Connects DTStat to the specified IP address using the specified port number instead of to the local machine
Examples
  • DTStat -i 300
  • DTStat -p -i 300 -t AlphaStats.sts
  • DTStat -f AlphaStats.sts -s AlphaStats.csv -start 02/02/2007 09:25
  • DTStat -server 206.31.4.51 1106
Notes
  • This command is not case-sensitive.
  • If no options are specified, DTStat will print the output to the screen at an interval of every one second.
  • If the statistics are not changing, DTStat will discontinue writing until statistics begin updating again.

Statistics

Category Statistic Description
Date/Time Stamp   The date and time that the snapshot was taken. This is the date and time that each statistic was logged. By default, these are generated once a second, as long as there are statistics being generated. If mirroring/replication is idle, then DTStat will be idle as well.
SystemAllocator Total Bytes The number of bytes currently allocated to the system pagefile
IQAllocator Total Bytes The number of bytes currently allocated to the intermediate queue
Security Logins The number of successful login attempts
FailedLogins The number of failed login attempts
Kernel SourceState
  • 0—Source is not running
  • 1—Source is running without the replication driver
  • 2—Source is running with the replication driver
TargetState
  • 0—Target is not running
  • 1—Target is running
Start Time Date and time stamp indicating when the Double-Take daemon was loaded
RepOpsGenerated The number of replication operations generated by the file system driver. An op is a file system operation. Double-Take Availability replicates data by sending the file system operations across the network to the target. RepOpsGenerated indicates the number of file system operations that have been generated by replication.
RepBytesGenerated The number of replication bytes generated by the file system driver. This is the number of bytes generated during replication. In other words, this is roughly the amount of traffic being sent across the network that is generated by replication. It does not take into account TCP/IP overhead (headers and such).
MirOpsGenerated The number of mirror operations transmitted to the target. Mirroring is completed by transmitting the file system operations necessary to generate the files on the target. This statistic indicates the number of file system operations that were transmitted during the initial mirror. It will continue to increase until the mirror is complete. Any subsequent remirrors will reset this field to zero and increment from there.
MirBytesGenerated The number of mirror bytes transmitted to the target. This is the number of bytes generated during mirroring. In other words, this is roughly the amount of traffic being sent across the network that is generated by the mirror. It does not take into account TCP/IP overhead (headers and such). Again, any subsequent remirror will reset this field to zero and increment from there.
FailedMirrorCount The number of mirror operations that failed due to an error reading the file from the disk
FailedRepCount The number of replication operations that failed due to an error reading the file from the disk
ActFailCount The number of activation code failures when loading the source or target. Activation codes can be bad for reasons such as: expiration of evaluation codes, duplicate codes, incorrect codes, etc.
TargetOpenHandles The number of handles currently open on the target
DriverQueuePercent The amount of throttling calculated as a percentage of the stop replicating limit
Target PeerAddress The IP address of the source machine
LocalAddress The IP address of the target machine.
Ops Received The total number of operations received by this machine as a target since the Double-Take daemon was loaded
Mirror Ops Received The total number of mirror operations received by this machine as a target since the Double-Take daemon was loaded. This number does not reset to zero for remirrors.
Retries The number of retries performed before all operations were completed
OpsDropped The number of operations skipped during a difference mirror. During a difference mirror, if Double-Take Availability detects that there have been no changes to a file, then it will indicate the number of operations it did not send for this file in this field.
Ops Remaining The total number of operations that are left in the target queue
Orphan Files Removed The number of orphan files removed from the target machine
Orphan Directories Removed The number of orphan directories removed from the target machine
Orphan Bytes Removed The number of orphan bytes removed from the target machine
Bytes In Target Queue The number of bytes currently in the system memory queue on the target
Bytes In Target Disk Queue The number of bytes currently in the disk queue on the target
TasksSucceeded The number of task commands that have succeeded on the target
TasksFailed The number of task commands that have failed on the target
TasksIgnored The number of task commands that have been ignored on the target
Source autoDisConnects The number of automatic disconnects since starting Double-Take Availability. Auto-disconnects occur because the source no longer sees the target This could be because the connection between the two has failed at some point or because the target machine data is changing on the source faster than the source can get the data to the target. This field tracks the number of times an auto-disconnect has occurred since the Double-Take daemon was started.
autoReConnects The number of automatic reconnects since starting Double-Take Availability. Auto-reconnect occurs after a target machine is back online. This field tracks the number of times an auto-reconnect has happened since the Double-Take daemon was started.
lastFileTouched The last filename that had a replication operation executed
Connection

 

conPeerAddress The IP address of the target machine
connectTime The time that this connection was established
conState

The state of the active connection

  • 0—None. This indicates a connection has not been established. Statistics are still available for the source and target machines.
  • 1—Active. This indicates that the connection is functioning normally and has no scheduling restrictions imposed on it at this time. (There may be restrictions, but it is currently in a state that allows it to transmit.)
  • 2—Paused. This indicates a connection that has been paused.
  • 4—Scheduled. This indicates a connection that is not currently transmitting due to scheduling restrictions (bandwidth limitations, time frame limitations, and so on).
  • 8—Error. This indicates a connection that is not transmitting because something has gone wrong (for example, lost connection).

Only the Scheduled and Error states can coexist. All other states are mutually exclusive. Statistics will display a conState of 12 when the connection is in both a scheduled and an error state because this is the sum of the two values (4 + 8).

conOpsInCmdQueue The number of operations waiting to be executed on the target
conOpsInAckQueue The number of operations waiting in the acknowledgement queue. Each operation that is generated receives an acknowledgement from the target after that operation has been received by the target. This statistic indicates the number of operations that have yet to receive acknowledgement of receipt.
conOpsInRepQueue The number of replication operations currently waiting to be executed on the target
conOpsInMirQueue The number of mirror operations currently waiting to be executed on the target
conBytesInRepQueue The number of replication bytes remaining to be transmitted to the target
conOpsTx The number of operations transmitted to the target. This is the total number of operations that Double-Take Availability has transmitted as a source. In other words, the cumulative number of operations transmitted by this source to all connected targets.
conBytesInMirQueue The number of mirror bytes remaining to be transmitted to the target
conBytesTx The number of bytes transmitted to the target. This is the total number of bytes that Double-Take Availability has transmitted as a source. In other words, the cumulative number of bytes transmitted by this source to all connected targets.
conBytesCompressedTx The number of compressed bytes transmitted to the target.
conOpsRx The number of operations received by the target. The number of operations that the target for this connection (as indicated by the IP address field) has received from this source.
conBytesRx The number of bytes received by the target. The number of bytes that the target for this connection (as indicated by the IP address field) has received from this source.
conResentOpCount The number of operations resent because they were not acknowledged
conBytesInDiskQueue The number of bytes in the source disk queue
conBandwidthLimit The amount of bandwidth that may be used to transfer data
conBytesSkipped The number of bytes skipped during a difference mirror. During a difference mirror, if Double-Take Availability detects that there have been no changes to a file, then it will indicate the number of bytes it did not send for this file in this field.
conMirrorBytesRemain The number of mirror bytes remaining to be transmitted
conMirrorPercent The percentage of the mirror that has been completed. This field is determined if the replication set size was calculated.
conTaskCmdsSubmitted The number of task commands that have been submitted on the source
conTaskCmdsQueued The number of task commands that have been queued on the source
conTasksSucceeded The number of task commands that have succeeded on the source
conTasksFailed The number of task commands that have failed on the source
conTasksIgnored The number of task commands that have been ignored on the source

SNMP

SNMP, Simple Network Management Protocol, is the Internet's standard for remote monitoring and management of hosts, routers and other nodes and devices on a network. Double-Take Availability provides an SNMP sub-agent that monitors Double-Take Availability and can be managed from an SNMP Management Console.

Double-Take Availability installs two components to work with SNMP.

Note: The Double-Take Availability SNMP sub-agent is only supported for NET -SNMP v2c.

Configuring SNMP on your server

SNMP must be installed, configured, and working on your server.

  1. Stop the SNMP daemon (snmpd).
  2. Make a backup copy of the SNMP configuration file snmpd.conf.
  3. In order to run the Double-Take Availability sub-agent, master agentx is needed to turn on agentx. Add the master agent line to snmpd.conf.

master agentx

  1. So that the SNMP daemon can locate the Double-Take Availability MIB, add the path to the Double-Take Availability MIB by adding an entry to the snmp.conf file.

mibfile /usr/share/snmp/mibs/NSI-DT-MIB.txt

  1. Restart the SNMP daemon (snmpd).
  2. Start the master agent.

#snmpd -f -Le -x /var/agentx/master &

  1. Start the Double-Take Availability SNMP sub-agent.

#DTSubAgent >& /dev/null &

Note: Instead of starting the master agent and Double-Take Availability sub-agent separately, you can add them both to init.d to start them automatically.
  1. You can test SNMP by trying either of the following commands.

#snmpget -v2c -c public localhost dtGeneral.dtUpTime.0

#snmpget -v2c -c public localhost NSI-MIB::dtUpTime

SNMP traps

The following table lists the Double-Take Availability SNMP traps.

Object Type Trap Description
Kernel dttrapKernelStarted Double-Take Availability has started
dttrapKernelStopped Double-Take Availability has stopped
License dttrapLicenseViolationStartingSource The source cannot be started due to a license violation
dttrapLicenseViolationOnNetwork A Double-Take Availability serial number conflict was identified on the network
Source dttrapSourceStarted Double-Take Availability source component has started
dttrapSourceStopped Double-Take Availability source component has stopped
Target dttrapTargetStarted Double-Take Availability target component has started
dttrapTargetStopped Double-Take Availability target component has stopped
Connection dttrapConnectionRequested The source has requested a connection to the target
dttrapConnectionRequestReceived The target has received a connection request from the source
dttrapConnectionSucceeded The source to target connection has been established
dttrapConnectionPause The source to target connection has paused
dttrapConnectionResume The source to target connection has resumed
dttrapConnectionFailed The source to target connection was not successful
dttrapConnectionLost The source to target connection has been disconnected
dttrapMemoryLimitReached The Double-Take Availability memory pool limit has been reached
dttrapMemoryLimitRemedied The memory pool usage is below the maximum limit specified
dttrapAutoReconnect Auto-reconnect needs to make a new connection
dttrapScheduledConnectStart A scheduled connection has been established
dttrapScheduledConnectEnd A scheduled end connection has been reached and the connection has been disconnected
dttrapAutoDisconnectWriteQueue Auto-disconnect has forced the queue to be written to disk
dttrapAutoDisconnectPauseTransmission Auto-disconnect requested that the source pause any operation (create, modify, or delete) sending
dttrapAutoDisconnectEndConnection Auto-disconnect has intentionally dropped the connection
dttrapAutoDisconnectShutdown Auto-disconnect forced Double-Take Availability to shutdown
Replication dttrapReplicationStart Replication has started
dttrapReplicationStop Replication has stopped
Mirroring dttrapMirrorStart Mirroring has started
dttrapMirrorStop Mirroring has stopped
dttrapMirrorPause Mirroring has paused
dttrapMirrorResume Mirroring has resumed
dttrapMirrorEnd Mirroring has ended
Verification dttrapVerificationStart Verification has started
dttrapVerificationEnd Verification has ended
dttrapVerificationFailure Verification has failed
Restoration dttrapRestoreStarted Restoration has started
dttrapRestoreComplete Restoration is complete
Replication Sets dttrapRepSetModified Replication has been modified
Failover dttrapFailoverConditionMet Manual intervention is required because failover has detected a failed source machine
dttrapFailoverInProgress Failover is occurring
dttrapTargetFull The target is full

SNMP statistics

The following table lists the Double-Take Availability SNMP statistics.

Object Type Statistic Description
General dtUpTime Time in seconds since Double-Take Availability was last started
dtCurrentMemoryUsage Amount of memory allocated from the Double-Take Availability memory pool
dtMirOpsGenerated The number of mirror operations (create, modify, or delete) that have been transmitted by the mirroring process
dtMirBytesGenerated The number of bytes that have been transmitted by the mirroring process
dtRepOpsGenerated The number of operations (create, modify, or delete) that have been transmitted by the replication process
dtRepBytesGenerated The number of bytes that have been transmitted by the replication process
dtFailedMirrorCount The number of operations that failed to mirror because they could not be read on the source
dtFailedRepCount The number of operations that failed to be replicated because they could not be read on the source
dtActFailCount The number of activation code errors
dtAutoDisCount The number of auto-disconnects
dtAutoReCount The number of auto-reconnects
dtDriverQueuePercent The amount of throttling calculated as a percentage of the stop replicating limit
Source dtSourceState
  • 0—Source is not running
  • 1—Source is running without the replication driver
  • 2—Source is running with the replication driver.
Target dtTargetState
  • 0—Target is not running
  • 1—Target is running
dtRetryCount The number of file operations that have been retried
dtOpsDroppedCount The number of file operations that have failed and will not be retried
Security dtLoginCount The number of successful logins
dtFailedLoginCount The number of unsuccessful logins
Connection dtConnectionCount The number of active connections between machines
dtconIpAddress The IP address of the connected machine. If at the source, then the IP address of the target. If at the target, then the IP address of the source.
dtconConnectTime The duration of time since the connection was first established
dtconState

The state of the active connection

  • 0—None. This indicates a connection has not been established. Statistics are still available for the source and target machines.
  • 1—Active. This indicates that the connection is functioning normally and has no scheduling restrictions imposed on it at this time. (There may be restrictions, but it is currently in a state that allows it to transmit.)
  • 2—Paused. This indicates a connection that has been paused.
  • 4—Scheduled. This indicates a connection that is not currently transmitting due to scheduling restrictions (bandwidth limitations, time frame limitations, and so on).
  • 8—Error. This indicates a connection that is not transmitting because something has gone wrong (for example, lost connection).

Only the Scheduled and Error states can coexist. All other states are mutually exclusive. SNMP will display a dtconState of 12 when the connection is in both a scheduled and an error state because this is the sum of the two values (4 + 8).

dtconOpsInCmdQueue The number of operations (create, modify, or delete) in the retransmit queue on the source
dtconOpsInAckQueue The number of operations (create, modify, or delete) waiting for verification acknowledgements from the target
dtconOpsInRepQueue The number of replication operations (create, modify, or delete) in the queue
dtconOpsInMirQueue The number of mirror operations (create, modify, or delete) in the queue
dtconBytesInRepQueue The number of bytes in the replication queue
dtconBytesInMirQueue The number of bytes in the mirror queue
dtconOpsTx The total number of operations (create, modify, or delete) transmitted to the target
dtconBytesTx The total number of bytes transmitted to the target
dtconBytesCompressedTx The total number of compressed bytes transmitted to the target
dtconOpsRx The total number of operations (create, modify, or delete) received from the target
dtconBytesRx The total number of bytes received from the target
dtconResentOpCount The number of operations that were resent because of acknowledgement errors

Connections

A unique connection ID is associated with each Double-Take Availability connection. The connection ID provides a reference point for each connection. The connection ID is determined by sequential numbers starting at one (1). Each time a connection is established, the ID counter is incremented. It is reset back to one each time the Double-Take daemon is restarted. For example, if the Double-Take daemon was started and the same replication set was connected to five target machines, each connection would have a unique connection ID from 1 to 5. The connection can be in various states as described in the following table.

Connection Status Description
Started

The network connection exists and is available for data transmission. Replication and mirror data are transmitted to the target as soon as possible. This is the standard state that you will see most often.

Stopped Double-Take Availability has linked the source and target, but the network connection does not exist. Replication and mirror data are not transmitted to the target but are held in queue on the source.
Paused

The network connection exists and is available for data transmission, but the replication and mirror data is being held in a queue and is not being transmitted to the target.

Scheduled

Double-Take Availability has linked the source and target, but the network connection is not established until event driven or scheduling criteria have been met.

Error

A transmission error has occurred. Possible errors include a broken physical line or a failed target daemon.

You can perform the following functions to manage your connections.

Data queues

During the Double-Take Availability installation, you identified the amount of disk space that can be used for Double-Take Availability queuing. Queuing to disk allows Double-Take Availability to accommodate high volume processing that might otherwise fill up system memory. For example, on the source, this may occur if the data is changing faster than it can be transmitted to the target, or on the target, a locked file might cause processing to backup.

The following diagram will help you understand how queuing works. Each numbered step is described after the diagram.

  1. If data cannot immediately be transmitted to the target, it is stored, or queued, in system memory. You can configure how much system memory you want to use for queuing. By default, 128 or 512 MB of memory is used, depending on your operating system.
  2. When the allocated amount of system memory is full, new changed data bypasses the full system memory and is queued directly to disk. Data queued to disk is written to a transaction log. Each transaction log can store 5 MB worth of data. Once the log file limit has been reached, a new transaction log is created. The logs can be distinguished by the file name which includes the target IP address, the Double-Take Availability port, the connection ID, and an incrementing sequence number.
  3. Note:

    You may notice transaction log files that are not the defined size limit. This is because data operations are not split. For example, if a transaction log has 10 KB left until the limit and the next operation to be applied to that file is greater than 10 KB, a new transaction log file will be created to store that next operation. Also, if one operation is larger than the defined size limit, the entire operation will be written to one transaction log.

  4. When system memory is full, the most recent changed data is added to the disk queue, as described in step 2. This means that system memory contains the oldest data. Therefore, when data is transmitted to the target, Double-Take Availability pulls the data from system memory and sends it. This ensures that the data is transmitted to the target in the same order it was changed on the source. Double-Take Availability automatically reads operations from the oldest transaction log file into system memory. As a transaction log is depleted, it is deleted. When all of the transaction log files are deleted, data is again written directly to system memory (step 1).
  5. To ensure the integrity of the data on the target, the information must be applied in the same order as it was on the source. If there are any delays in processing, for example because of a locked file, a similar queuing process occurs on the target. Data that cannot immediately be applied is queued to system memory. By default, 128 or 512 MB of memory is used, depending on your operating system.
  6. When the allocated amount of system memory on the target is full, new incoming data bypasses the full system memory and is queued directly to disk. Data queued to disk is written to a transaction log. On the target, the transaction logs are identified with the source IP address, the Double-Take Availability port, the connection ID, and an incrementing sequence number.

Like the source, system memory on the target contains the oldest data so when data is applied to the target, Double-Take Availability pulls the data from system memory. Double-Take Availability automatically moves operations from the oldest transaction log file to system memory. As a transaction log is depleted, it is deleted. When all of the transaction log files are deleted, data is again written directly to system memory (step 4).

Queuing data

You should configure queuing on both the source and target.

  1. Right-click the server on the left pane of the Replication Console.
  2. Select Properties.
  3. Select the Queue tab.
  4. Specify the queue settings for the server.

Select a location on a volume that will have minimal impact on the operating system and applications being protected. For best results and reliability, this should be a dedicated, non-boot volume. The disk queue should not be on the same physical or logical volume as the data being replicated.

Note:

Scanning the Double-Take Availability queue files for viruses can cause unexpected results. If anti-virus software detects a virus in a queue file and deletes or moves it, data integrity on the target cannot be guaranteed. As long as you have your anti-virus software configured to protect the actual production data, the anti-virus software can clean, delete, or move an infected file and the clean, delete, or move will be replicated to the target. This will keep the target from becoming infected and will not impact the Double-Take Availability queues.

Since the source is typically running a production application, it is important that the amount of memory Double-Take Availability and the other applications use does not exceed the amount of RAM in the system. If the applications are configured to use more memory than there is RAM, the system will begin to swap pages of memory to disk and the system performance will degrade. For example, by default an application may be configured to use all of the available system memory when needed, and this may happen during high-load operations. These high-load operations cause Double-Take Availability to need memory to queue the data being changed by the application. In this case, you would need to configure the applications so that they collectively do not exceed the amount of RAM on the server. Perhaps on a server with 1 GB of RAM running the application and Double-Take Availability, you might configure the application to use 512 MB and Double-Take Availability to use 256 MB, leaving 256 MB for the operating system and other applications on the system. Many server applications default to using all available system memory, so it is important to check and configure applications appropriately, particularly on high-capacity servers.

Any changes to the memory usage will not take effect until the Double-Take daemon has been restarted on the server.

  1. Click OK to save the settings.

Auto-disconnect and auto-reconnect

While disk queues are user configurable and can be extensive, they are limited. If the amount of disk space specified for disk queuing is met, additional data could not be added to the queue and data would be lost. To avoid any data loss, the auto-disconnect and auto-reconnect processes occur.

Reconnecting automatically

Use the following steps to configure automatic reconnections.

  1. Right-click the source server on the left pane of the Replication Console and select Properties.
  2. Select the Setup tab.

  1. Verify that the check box Automatically Reconnect During Source Initialization is marked to enable the auto-reconnect feature.
  2. Click OK to save the settings.

Pausing and resuming target processing

You can break the source/target connection without disconnecting the connection, so that you can control the transmission of data across the network. You can do this by pausing the target. If the target is paused, data is queued on the source until you manually resume the target. For example, you may want to pause the target while you perform a backup of the target data, and then resume the target when the backup is complete.

While the target is paused, the Double-Take Availability source cannot queue data indefinitely. If the source queue is filled, Double-Take Availability will automatically disconnect the connections and attempt to reconnect them.

To pause a target, right-click a target server on the left pane of the Replication Console and select Pause Target. All active connections to that target will complete the operations already in progress. You will see Pause Pending in the Replication Console while these operations are completed. The status will update to Paused after the operations are completed. Any new operations will be queued on the source until the target is resumed. When you are ready to resume the target, right-click the target and select Resume Target.

Note: If you have multiple connections to the same target, all connections will be paused and resumed.

Disconnecting a connection

To disconnect a Double-Take Availability connection, right-click the connection on the right pane of the Replication Console and select Disconnect. The source and target will be disconnected.

Note:

If a connection is disconnected and the target is monitoring the source for failover, you will be prompted if you would like to continue monitoring for a failure. If you select Yes, the Double-Take Availability connection will be disconnected, but the target will continue monitoring the source. To make modifications to the failure monitoring, you will need to use the Failover Control Center. If you select No, the Double-Take Availability connection will be disconnected, and the source will no longer be monitored for failure by the target.

If a connection is disconnected while large amounts of data still remain in queue, the Replication Console may become unresponsive while the data is being flushed. The Replication Console will respond when all of the data has been flushed from the queue.

Mirroring

Mirroring is one of the key components of Double-Take Availability. You can perform the following functions to manage mirroring.

Stopping, starting, pausing, or resuming mirroring

After a connection is established, you need to be able to control the mirroring. You can start, stop, pause and resume mirroring. Right-click the connection on the right pane of the Replication Console and select Mirroring and the appropriate mirror control.

File difference mirror options compared

An X enclosed in parentheses (X) indicates that the global option on the Server Properties Source tab can be on or off. The use of this option does not change the action performed during the mirror.

Mirroring automatically

In certain circumstances, for example if the disk-based queues on the source are exhausted, Double-Take Availability will automatically disconnect connections (called auto-disconnect) and then automatically reconnect them (called auto-reconnect). In order to ensure data integrity on the target, Double-Take Availability will perform an automatic mirror (called an auto-remirror) after an auto-reconnect.

Note:

Auto-remirror is a per source option. When enabled, all connections from the source will perform an auto-remirror after an auto-reconnect. When disabled, none of the connections from the source will perform an auto-remirror after an auto-reconnect.

  1. Right-click a server in the left pane of the Replication Console and select Properties.
  2. Select the Setup tab.

  1. Verify that the Perform Remirror After Auto-Reconnect check box is selected to initiate an auto-remirror after an auto-reconnect.
  2. Note: If auto-remirror is disabled and an auto-reconnect occurs, the transmission state of the connection will remain pending after the reconnect until a mirror is started manually.
  3. Specify the type of mirror that you wish to perform.
  1. Click OK to save the settings.

Removing orphan files

An orphan file is a file that exists in the target’s copy of the replication set data, but it does not exist in the source replication set data. An orphan file can be created when you delete a file contained in the source replication set while there is no Double-Take Availability connection. For example, if a connection was made and a mirror was completed and then the connection was stopped and a file was deleted on the source, an orphan file will exist on the target. Because the connection has been disconnected, the delete operation is not replicated to the target and the file is not deleted on the target. Additionally, orphan files may also exist if files were manually copied into or deleted from the location of the target’s copy of the replication set data.

You can configure orphan files to be moved or deleted automatically during a mirror, verify, or restore, or you can move or delete orphan files manually at any time. You can move or delete all orphan files on the target or only those orphan files that are older than a specified period of time. The results of orphan processing are maintained in the Double-Take Availability log on the target, including the number of moved/deleted orphan files, the directories, and the number of bytes.

Note:

Orphan file configuration is a per target option. All connections to the same target will have the same orphan file configuration.

The orphans feature does not move or delete alternate data streams. To do this, use a full mirror which will delete the additional stream(s) when the file is re-created.

If Double-Take Availability is configured to move orphan files, the Double-Take Availability log file will indicate that orphan files have been deleted even though they have actually been moved. This is a reporting issue only.

If delete orphans is enabled, directories and files that do not exist on the source and are excluded in the replication set using a wildcard rule will be removed from the target path. If you have data in your target path that does not exist on the source, do not use wildcard rules in your replication set. Manually select and deselect those files which should be included or excluded from your replication set.

  1. If you want to remove orphan files manually, right-click an established connection and select Remove Orphans, Start.
  2. If you want to stop the process after it has been started, right-click the connection and select Remove Orphans, Stop.
  3. To configure orphan files for processing during a mirror, verify, or restore, use the following instructions.
  1. Right-click the connection on the right pane of the Replication Console and select Connection Manager.
  2. Select the Orphans tab.

  1. Specify if you want to log the name of the orphan files to the Double-Take Availability log file on the target by marking Log Orphaned Files to Target Log.
  2. By default, the orphan files feature is disabled. To enable it, mark Move/Delete Orphan Files.
  3. Specify if you want to Delete Orphaned Files or Move Orphaned Files to a different location. If you select the move option, identify the location where these orphan files will be located.
  4. Note:

    If you are moving files, make sure the directory you specify to move the files to is not included in the destination of the replication set data so that the orphan files are only moved once.

  5. Specify if you want to Remove All Orphans or Remove Orphans not modified within the following time period. If you select the time-based option, only orphans older than the time you specify will be removed.
  6. Click OK to save the settings.

Replication

Replication is one of the key components of Double-Take Availability. This section contains the following replication topics.

Replication capabilities

Double-Take Availability replicates all file and directory data in the supported Linux file systems. Double-Take Availability does not replicate items that are not stored on the file system, such as pseudo-file systems like /proc and /sys. In addition, note the following.

If you want to make sure all DTFS mount points are accounted for, view the list from the start and stop daemon menu in DTSetup or by doing a cat of /etc/DT/dtfs_mounts.

When managing these mount points manually, the lower or subordinate native mount points must not be mounted until DTFS is mounted for the parent mount. Using the same example and starting with all native volumes mounted, the following steps would be required.

  1. umount /level1/level2/level3
  2. umount /level1/level2/
  3. mount dtfs –t dtfs –o norepatime /level1
  4. mount /dev/sda2 –t ext3 –o noatime /level1/level2
  5. mount dtfs –t dtfs –o norepatime /level1/level2
  6. mount /dev/sda3 –t ext3 –o noatime /level1/level2/level3
  7. mount dtfs –t dtfs –o norepatime /level1/level2/level3

Replication sets

A replication set defines the data on a source machine that Double-Take Availability protects. Replication sets are defined by volumes, directories, files, or wild card combinations. Creating multiple replication sets allows you to customize sets of data that need to be protected.

When a replication set is created, a series of rules are defined that identify the volumes, directories, files, and/or wild card combinations that will be replicated to the target. Each rule includes:

For example, a replication set rule might be volume\directory\* inc, rec

This specifies that all files contained in the volume\directory path are included in the replication set. Because recursion is set, all files and subdirectories under volume\directory are also included. A complete replication set becomes a list of replication set rules.

Replication sets offer flexibility tailoring Double-Take Availability to your environment. For example, multiple replication sets can be created and saved for a source to define a unique network configuration. There may be three replication sets - Critical Data, User Data, and Offsite Data. Critical Data could be configured to replicate, in real-time, to an onsite high-availability server. Offsite Data is replicated across a WAN and, therefore, is configured to queue changes until a sufficient amount of data is changed to justify transmission. At that point, the connection is made and stays active until all the data is transmitted. User Data is not replicated throughout the day, but a nightly changed file mirror copies only blocks of data that are different between the source and target server prior to a nightly tape backup operation being run on the target server. Each of these replication sets can be automated to transmit as needed, thus protecting your entire environment.

Keep in mind the following notes when creating and working with replication sets and connections.

Creating a replication set

Before you can establish a connection, you must create a replication set.

  1. Highlight a source in the left pane of the Replication Console and select Insert, Replication Set from the menu bar. You can also right-click on the source name and select New, Replication Set.
  2. A replication set icon appears in the left pane under the source. By default, it is named New Replication Set. Rename the newly inserted replication set with a unique name by typing over the default name and pressing Enter. This process is similar to naming a new folder in Windows Explorer.
  3. Expand the tree under the replication set name to view the volume and directory tree for the source.
  4. Note:

    The default number of files that are listed in the right pane of the Replication Console is 2500, but this is user configurable. A larger number of file listings allows you to see more files in the Replication Console, but results in a slower display rate. A smaller number of file listings displays faster, but may not show all files contained in the directory. To change the number of files displayed, select File, Options and adjust the File Listings slider bar to the desired number.

    To hide offline files, such as those generated by snapshot applications, select File, Options and disable Display Offline Files. Offline files and folders are denoted by the arrow over the lower left corner of the folder or file icon.

  1. Identify the data on the source that you want to protect by selecting volumes, drives, directories, and/or specific files.

Note: Be sure and verify what files can be included by reviewing Replication capabilities.
  1. After selecting the data for this replication set, right-click the new replication set icon and select Save. A saved replication set icon will change from red to black.
  2. If you need to select a block device for replication, right-click the replication set and select Add Device.
  3. The block devices configured for Double-Take Availability replication are shown by default. Highlight the device to include in the replication set and click OK.
  4. Note: If the device you want to include is not displayed, you can click Show Other Devices to view all devices which are eligible for Double-Take Availability replication. You can select any of these devices, but you cannot use them for Double-Take Availability replication until they are configured for Double-Take Availability replication. The status no dtloop indicates the device is not configured for Double-Take Availability replication.
  5. Repeat steps 6 and 7 for any additional devices.
  6. Right-click the updated replication set icon and select Save.

Creating or modifying replication rules manually

There may be times when you cannot browse for data when creating a replication set. For example, you can create a replication set rule for a directory or file that does not exist. Since you cannot browse for the location, you have to create replication set rule manually. At other times, the data you want to replicate cannot be easily selected from the Replication Console. For example, you may want to select all .db files from a specific volume or directory. This task may be easier to complete by creating the replication set rule manually. Use the following instructions to create or modify a replication set rule manually.

  1. If you do not have a replication set created, you need to create one. Highlight a source in the left pane of the Replication Console and select Insert, Replication Set from the menu bar. You can also right-click on the source name and select New, Replication Set. A replication set icon appears in the left pane under the source. By default, it is named New Replication Set. Rename the newly inserted replication set with a unique name by typing over the default name and pressing Enter. This process is similar to naming a new folder in Windows Explorer.
  2. Right-click on the replication set icon and select Properties. The Replication Set Properties dialog box appears and lists any existing rules. The existing rules may have been entered manually or selected by browsing the source. Each rule will display the attributes associated it.

  1. From the Replication Set Properties dialog box, click Add.
  2. Specify a path, wild card, or specific file name. Select the Include, Exclude, and/or Recurse sub-directories attributes to be applied to this rule and click OK.
  3. If you need to select block devices for replication, click Add Device. The block devices configured for Double-Take Availability replication are shown by default. Highlight the device to include in the replication set and click OK. If the device you want to include is not displayed, you can click Show Other Devices to view all devices which are eligible for Double-Take Availability replication. You can select any of these devices, but you cannot use them for Double-Take Availability replication until they are configured for Double-Take Availability replication. The status no dtloop indicates the device is not configured for Double-Take Availability replication.
  4. If you need to edit an existing rule, highlight it and click Edit.
  5. If you need to remove a rule, highlight it and click Remove.
  6. After the replication set rules have been defined, exit the Replication Set Properties dialog box by clicking OK. Notice the replication set icon has changed from black to red, indicating changes to the replication set rules. If you click Cancel, your changes will not be reflected in the current replication set.
  7. Right-click the replication set icon and select Save. A saved replication set icon will change from red to black.

Selecting a block device for replication

Double-Take Availability allows you to select block devices for replication.

  1. In the left pane, right-click the replication set that should include the block device and select Add Device.
  2. The block devices configured for Double-Take Availability replication are shown by default. Highlight the device to include in the replication set and click OK.
  3. Note: If the device you want to include is not displayed, you can click Show Other Devices to view all devices which are eligible for Double-Take Availability replication. You can select any of these devices, but you cannot use them for Double-Take Availability replication until they are configured for Double-Take Availability replication. The status no dtloop indicates the device is not configured for Double-Take Availability replication.
  4. Repeat steps 1 and 2 for any additional devices.

Modifying a replication set

Double-Take Availability allows you to make modifications to a replication set when you want to change the data you wish to protect. This allows you to add, remove, or modify any replication set rules without having to create a new replication set.

  1. In the left pane, highlight the replication set you want to modify and expand the volume and directory levels as needed.
  2. Modify the items by marking or clearing the volume, drive, directory, or file check boxes. Notice the replication set icon has changed from black to red, indicating changes to the replication set rules.
  3. After updating the rules for this replication set, right-click the replication set icon and select Save. A saved replication set icon will change from red to black.
  4. Note:

    If you save changes to a connected replication set, it is recommended that you perform a mirror to guarantee data integrity between the source and target machines. A dialog box will appear instructing you to disconnect and reconnect the replication set and perform a difference mirror.

Renaming and copying a replication set

To rename or copy a replication set, click once on a highlighted replication set name to edit the field. Specify a unique name and press Enter. The process is similar to renaming a folder in Windows Explorer. If the original replication set has not been saved (red icon), the new name replaces the original name. If the original replication set is saved (black icon), the new name creates a copy of the original replication set.

Note:

If you save changes to a connected replication set, it is recommended that you perform a mirror to guarantee data integrity between the source and target machines. A dialog box will appear instructing you to disconnect and reconnect the replication set and perform a difference mirror.

Calculating replication set size

While Double-Take Availability is mirroring, the right pane of the Replication Console displays statistics to keep you informed of its progress. If the size of the replication set is determined before the mirror is started, Double-Take Availability can display the percentage of the replication set that has been mirrored in the Mirror Status column. If the size was not calculated prior to starting the mirror, the column displays Mirroring.

  1. Right-click on the replication set icon and select Properties. The Replication Set Properties dialog box appears.
  2. Select the Size tab.

  1. If the replication set size has never been determined, click Calculate. If the replication set has previously been determined, the button will be labeled Recalculate. Depending on user activity, the size shown may not accurately reflect the current size of the replication set. If changes are occurring to files in the replication set while the calculation is being made, the actual size may differ slightly. The amount of data is determined at the exact time the calculation is made.
  2. Click OK to return to the Replication Console.
  3. Note:

    You can also configure the replication set calculation when establishing a connection through the Connection Manager by selecting Calculate Replication Set size on connection on the Mirroring tab.

    If your replication set contains a large number of files, for example, ten thousand or more, you may want to disable the calculation of the replication set size so that data will start being mirrored sooner. If calculation is enabled, the source calculates the file size before it starts mirroring. This can take a significant amount of time depending on the number of files and system performance. Disabling calculation will result in the mirror status not showing the percentage complete or the number of bytes remaining to be mirrored.

Exporting and importing a replication set

To help reuse replication sets between servers, you can export an existing replication set on one server and import it on another.

Exporting a replication set
  1. Right-click an existing replication set and select Export.
  2. Select a location and file name for the replication set information, and click Save. If you want to share the replication set information with other consoles, select a location accessible by other consoles.
Importing a replication set
  1. Right-click the server where you want to import the replication set and select New, Import Replication Set.
  2. Locate the replication set information file and click Open.
  3. By default, the new replication set will have the same name as the original replication set. If desired, modify the name. Press Enter to accept the replication set name.
  4. By default, the new replication set is imported in an unsaved state. An unsaved replication set icon is red. Modify the replication set definition (include or exclude volumes or files) and then save the replication set by right-clicking on it and selecting Save. A saved replication set icon is black.

 

Deleting a replication set

You can only delete a replication set if it is not currently connected. If the replication set is connected, you must disconnect the connection and then delete the replication set.

To delete a replication set, right-click the replication set icon and select Delete. Additionally, you can highlight the replication set and press the Delete key on the keyboard.

Starting replication

Starting replication when establishing a connection is the default and recommended configuration. If replication is not started, data is not added to the queue on the source, and source/target data integrity is not guaranteed.

To start replication, right-click the connection on the right pane of the Replication Console and select Replication, Start. After starting replication, you should perform a remirror to guarantee the source and target data are identical.

Inserting tasks during replication

Task command processing is a Double-Take Availability feature that allows you to insert and run tasks at various points during the replication of data. Because the tasks are user-defined, you can achieve a wide variety of goals with this feature. For example, you might insert a task to create a snapshot or run a backup on the target after a certain segment of data from the source has been applied on the target. This allows you to coordinate a point-in-time backup with real-time replication.

Task command processing can be enabled from the Replication Console, but it can only be initiated through the scripting language. See the Scripting Guide for more information.

To enable task command processing from the Replication Console, right-click a server in the left pane of the Replication Console, select Properties, select the Setup tab, and select Enable Task Command Processing.

Note: If you disable this option on a source server, you can still submit tasks to be processed on a target, although task command processing must be enabled on the target.

Verification

Verification is the process of confirming that the data on the target is identical to the data on the source. Verification creates a log file detailing what was verified as well as which files are not synchronized. If the data is not the same, Double-Take Availability can automatically initiate a remirror. The remirror ensures data integrity between the source and target.

Verifying manually

A manual verification can be run anytime a mirror is not in progress.

  1. Right-click the connection on the right pane of the Replication Console and select Verify.
  2. Select the verification options that you would like to perform.

  1. Click OK to start the verification.

Verifying on a schedule

Verification can be scheduled to occur automatically at periodic intervals.

  1. Right-click the connection on the right pane of the Replication Console and select Connection Manager.
  2. Select the Verify tab.

  1. Specify when you want to start the initial verification. Select the immediate date and time by clicking Now, or enter a specific Date and Time. The down arrow next to Date displays a calendar allowing easy selection of any date. Time is formatted for any AM or PM time.
  2. Mark the Reverification Interval check box to repeat the verification process at the specified interval. Specify an amount of time and choose minutes, hours, or days.
  3. Select if you want to Remirror data to the target automatically. When enabled, Double-Take Availability will verify the data, generate a verification log, and remirror to the target any files that are different on the source. If disabled, Double-Take Availability will verify the data and generate a verification log, but no files will be remirrored to the target.
  4. If you are remirroring your files, you can specify Only send data if source’s date is newer than the target’s date so that only files that are newer on the source than on the target are remirrored.
  5. Note:

    If you are using a database application, do not use the newer option unless you know for certain you need it. With database applications, it is critical that all files, not just some of them that might be newer, get mirrored.

  1. Specify if you want the verification process to Use Checksum to send minimal blocks of data to determine which blocks are different. If this option is enabled, only those blocks (not the entire files) that are different will be identified in the log and/or remirrored to the target.
  2. Note:

    Database applications may update files without changing the date, time, or file size. Therefore, if you are using database applications, you should use the block checksum comparison to ensure proper verification and remirroring.

  1. Click OK to save the settings.

Configuring the verification log

A verification log is created on the source during the verification process. The log identifies what is verified as well as which files are not synchronized.

  1. Right-click the source server on the left pane of the Replication Console and select Properties.
  2. Select the Logging tab.

  1. At the top of the window, Folder identifies the location where the log files identified on this tab are stored. By default, the log files are stored in the same directory as the Double-Take Availability program files.
  2. Under the Verification section, Filename contains the base log file name for the verification process. The replication set name will be prepended to the base log file name. For example, since the default is DTVerify.log, the verification log for the replication set called UserData would be UserData DTVerify.log.
  3. Specify the Maximum Length of the log file. The default is 1048576 bytes (1 MB). When the log file reaches this limit, no additional data will be logged.
  4. By default, the log is appended to itself each time a verification process is completed. Clear the Append check box if you do not want to append to the previous log file.
  5. Note: Changes made to the verification log in the Server Properties, Logging tab will apply to all connections from the current source machine.
  1. Specify the Language of the log file. Currently, English is the only available language.
  2. Click OK to save the settings.

In the log file, each verification process is delineated by beginning and end markers. A list of files that are different on the source and target is provided as well cumulative totals for the verification process. The information provided for each file is the state of its synchronization between the source and the target at the time the file is verified. If the remirror option is selected so that files that are different are remirrored, the data in the verify log reflects the state of the file before it is remirrored, and does not report the state of the file after it is remirrored. If a file is reported as different, review the output for the file to determine what is different.

Data transmission

Double-Take Availability data is continuously transmitted to the target machine. Although the data may be queued if the network or target machine is slow, the default transmission setting is to transmit the data as soon as possible. You can modify the transmission to suit your environment.

Stopping, starting, pausing, and resuming transmission

To start, pause, or resume the transmission of data from the source to the target, right-click an established connection and select Transmit and the appropriate transmission control.

Scheduling data transmission

Using the Connection Manager Transmit tab, you can set start and stop criteria along with a schedule window.

Note:

Double-Take Availability checks the schedule once every second, and if a user-defined criteria is met, transmission will start or stop, depending on the option specified.

Any replication sets from a source connected to the same IP address on a target will share the same scheduled transmission configuration.

  1. Right-click the connection on the right pane of the Replication Console and select Connection Manager.
  2. Select the Transmit tab. The Transmit tab contains four limit types: Bandwidth, Start, Stop, and Window. The transmission options for each limit type are displayed by highlighting a selection in the Limit Type box.

At the top of the Transmit tab dialog box, the Enable Transmission Limiting check box allows you to turn the transmission options on or off. You can enable the transmission options by marking the Enable Transmission Limiting check box when you want the options to be applied, but you can disable the transmission options, without losing the settings, by clearing that check box.

Also at the top of the Transmit tab dialog box, the Clear All button, when selected, will remove all transmission limitations that have been set under any of the limit types. The Clear button will clear the settings only for the Limit Type selected.

  1. When you schedule transmission start criteria, transmission will start when the criteria is met and will continue until the queue is empty or a transmission stop criteria is met. Select the Start option in the Limit Type box.

Define the start options for Double-Take Availability transmission by using any combination of the following options.

  1. Schedule any desired stop criteria to stop transmission after a transmission start criteria has initiated the transmission. If you do not establish a stop criteria, transmission will end when the queue is empty. Select the Stop option in the Limit Type box.

Define the stop options to stop Double-Take Availability transmissions by using either or both of the following options.

  1. Schedule a transmission window to establish a period of availability for all Double-Take Availability transmissions. You can specify a begin and end time for all Double-Take Availability transmissions. When a transmission window is in effect, all other start and stop criteria are bound by this window. This means that Double-Take Availability will never transmit data outside of an established window, regardless of other transmission settings. For example, if you set a window of availability from 9 p.m. to 4 a.m. and a start option to initiate transmission at 5 a.m., the window option will override the start option and no data will be sent at 5 a.m. Select the Window option in the Limit Type box.
  2. Note:

    Setting a transmission window by itself is not sufficient to start a transmission. You still need to set a start criteria within the window.

Define a window to control Double-Take Availability transmissions by enabling the feature and then specifying both window options.

  1. Click OK to save the settings.

Limiting transmission bandwidth

Using the Connection Manager Transmit tab, you can set start and stop criteria along with a schedule window.

Note:

Double-Take Availability checks the schedule once every second, and if a user-defined criteria is met, transmission will start or stop, depending on the option specified.

Any replication sets from a source connected to the same IP address on a target will share the same scheduled transmission configuration.

  1. Right-click the connection on the right pane of the Replication Console and select Connection Manager.
  2. Select the Transmit tab. The Transmit tab contains four limit types: Bandwidth, Start, Stop, and Window. The transmission options for each limit type are displayed by highlighting a selection in the Limit Type box.

At the top of the Transmit tab dialog box, the Enable Transmission Limiting check box allows you to turn the transmission options on or off. You can enable the transmission options by marking the Enable Transmission Limiting check box when you want the options to be applied, but you can disable the transmission options, without losing the settings, by clearing that check box.

Also at the top of the Transmit tab dialog box, the Clear All button, when selected, will remove all transmission limitations that have been set under any of the limit types. The Clear button will clear the settings only for the Limit Type selected.

  1. Select the Bandwidth option in the Limit Type box. Mark the Limit Bandwidth check box to enable the bandwidth limiting features. Define the bandwidth available for Double-Take Availability transmission by using either of the following options.

  1. Click OK to save the settings.

Compressing data for transmission

To help reduce the amount of bandwidth needed to transmit Double-Take Availability data, compression allows you to compress data prior to transmitting it across the network. In a WAN environment this provides optimal use of your network resources. If compression is enabled, the data is compressed before it is transmitted from the source. When the target receives the compressed data, it decompresses it and then writes it to disk. On a default Double-Take Availability installation, compression is disabled.

Note:

Any replication sets from a source connected to the same IP address on a target will share the same compression configuration.

Keep in mind that the process of compressing data impacts processor usage on the source. If you notice an impact on performance while compression is enabled in your environment, either adjust to a lower level of compression, or leave compression disabled. Use the following guidelines to determine whether you should enable compression:

  1. Right-click the connection on the right pane of the Replication Console and select Connection Manager.
  2. Select the Compression tab.
  3. By default, compression is disabled. To enable it, select Enable Compression.
  4. Depending on the compression algorithms available for your operating system, you may see a slider bar indicating different compression levels. Set the level from minimum to maximum compression to suit your needs.
  5. Click OK to save the settings.

Failover and failback

Failover is the process in which a target stands in for a failed source. As a result, user and application requests that are directed to the failed source are routed to the target.

Double-Take Availability monitors the source status by tracking network requests and responses exchanged between the source and target. When a monitored source misses a user-defined number of requests, Double-Take Availability assumes that the machine has failed. Double-Take Availability then prompts the network administrator to initiate failover, or, if configured, it occurs automatically.

The failover target assumes the network identity of the failed source. When the target assumes the identity of the source, user and application requests destined for the source machine or its IP address(es) are routed to the target.

When partnered with the Double-Take Availability data replication capabilities, failover routes user and application requests with minimal disruption and little or no data loss. In some cases, failover may be used without data replication to ensure high availability on a machine that only provides processing services, such as a web server.

Failover can be configured to stand in for one or more IP addresses associated with different NICs on the source. Each IP address can be added to a specific target NIC making NIC configuration very flexible. For example, a single NIC on the source may have one or more IP addresses assigned to it. If that source or the NIC fails, all traffic from the source is directed to the target. If there are multiple NICs on the source, the target can assume the traffic from all of the addresses. Additional NICs on the target increase flexibility and control. Secondary target NICs can assume the traffic from a failed source NIC while normal target traffic can continue to use the primary target NIC.

Failback is the process in which the target releases the source identity so that the source can be brought back onto the network.

Configuring failover monitoring

Delete this text and replace it with your own content.

  1. The Failover Control Center can be started from within the Replication Console or from the Windows desktop.
  1. Select a failover target from the Target Machine list box.
  2. Note:

    If the target you need is not listed, click Add Target and manually enter a name or IP address (with or without a port number). You can also select the Browse button to search for a target machine name. Click OK to select the target machine and return to the Failover Control Center main window.

  3. Click Login to login to the selected target.
  4. Select a source machine to monitor by clicking Add Monitor. The Insert Source Machine dialog box appears in front of the Monitor Settings dialog box.
  5. On the Insert Source Machine dialog, specify your source machine by any of the following methods.

The Insert Source Machine dialog closes and the Monitor Settings dialog remains open with your source listed in the Names to Monitor tree.

  1. In the Names to Monitor tree, locate and select the IP addresses on the source that you want to monitor.
  2. Highlight an IP address that you have selected for monitoring and select a Target Adapter that will assume that IP address during failover. Repeat this process for each IP address that is being monitored.
  3. Note:

    Current IP Addresses displays the IP address(es) currently assigned to the selected target adapter.

  4. Highlight an IP address that you have selected for monitoring and select a Monitor Interval. This setting identifies the number of seconds between the monitor requests sent from the target to the source to determine if the source is online. Repeat this step for each IP address that is being monitored.
  5. Highlight an IP address that you have selected for monitoring and select the Missed Packets. This setting is the number of monitor replies sent from the source to the target that can be missed before assuming the source machine has failed. Repeat this step for each IP address that is being monitored.
  6. Note:

    To achieve shorter delays before failover, use lower Monitor Interval and Missed Packets values. This may be necessary for IP addresses on machines, such as a web server or order processing database, which must remain available and responsive at all times. Lower values should be used where redundant interfaces and high-speed, reliable network links are available to prevent the false detection of failure. If the hardware does not support reliable communications, lower values can lead to premature failover. To achieve longer delays before failover, choose higher values. This may be necessary for IP addresses on slower networks or on a server that is not transaction critical. For example, failover would not be necessary in the case of a server restart.

  1. Highlight the source name and specify the Items to Failover, which identifies which source components you want to failover to the target.
  1. By default, Manual Intervention is enabled, allowing you to control when failover occurs. When a failure occurs, a prompt appears in the Failover Control Center and waits for you to manually initiate the failover process. Disable this option only if you want failover to occur immediately when a failure occurs.
  2. If you are using any failover or failback scripts, click Scripts and enter the path and filename for each script type. Scripts may contain any valid Linux command, executable, or script file. Examples of functions specified in scripts include stopping daemons on the target before failover because they may not be necessary while the target is standing in for the source, stopping daemons on the target that need to be restarted with the source’s machine name and IP address, starting daemons or loading applications that are in an idle, standby mode waiting for failover to occur, notifying the administrator before and after failover or failback occurs, stopping daemons on the target after failback because they are no longer needed, stopping daemons on the target that need to be restarted with the target machine’s original name and IP address, and so on.Specify each script that you want to run and the following options, if necessary.
  1. Click OK to return to the Monitor Settings dialog box.
  2. Click OK on the Monitor Settings dialog box to save your monitor settings and begin monitoring for a failure.

WAN considerations

When the source and target are on the same subnet, generally a LAN environment, you should failover the IP address. However, if the source and target are on different subnets, generally a WAN environment, you should not failover the IP address. You have several options for handling WAN failover.

Scripting example using the BIND DNS client
  1. Install the BIND DNS client on the target server, if it is not already installed.
  2. Create a PATH statement on the target for the BIND directory to ensure that it runs every time the executable is called.
  3. Update the Double-Take daemon on the target to use a domain account that has rights to modify BIND DNS. You will have to stop and restart the daemon for changes to the user account to take effect.
  4. Create a failover script with the following command. Specify this script for post-failover.
  5. nsupdate "dnsover"

  6. Create a file called dnsover and add the following lines. This is the file called by your post-failover script.
  7. # Substitute your source name, target name, and target IP address

     

    update delete source_server_name.fully_qualified_domain.com

    update add target_server_name.fully_qualified_domain.com 86400 A target_server_IP

    send

  8. Create a failback script with the following command. Specify this script for post-failback.
  9. nsupdate "dnsback"

  10. Create a file called dnsback and add the following lines. This is the file called by your post-failback script.
  11. # Substitute your target name, source name, and source IP address

    update delete target_server_name.fully_qualified_domain.com A

    update add source_server_name.fully_qualified_domain.com 86400 A source_server_IP

    send

When failover and failback occur, the failover and failback scripts will automatically trigger DNS updates.

 

Protecting NFS shares

A share is any local volume, drive, or directory resource that is exported and shared across a network. NFS shares must be configured for failover through the failover scripts or created manually on the target.

  1. Mount DTFS on the directory on the source where the NFS share is located.
  2. Start the Double-Take daemon on the source.
  3. Stop and restart the NFS daemon on the source. The Double-Take daemon must be running before the NFS daemon in order for replication operations to be captured.
  4. On your target, set the NFS daemon to manual startup. This allows the failover script to control when the daemon starts on the target.
  5. Create a replication set on the source that includes /etc/exports and the shared data that has DTFS mounted.
  6. Connect the replication set using the Connection Wizard or the Connection Manager.
  7. Add the following to your post-failover script.
  8. service nfs start

  9. If necessary, update DNS.

After failover, the NFS daemon will automatically be started by the post-failover script. If your clients see a stale file handle error message when attempting to access a share, they will need to reconnect to the share.

 

Protecting Samba shares

A share is any local volume, drive, or directory resource that is exported and shared across a network. Samba shares must be configured for failover through the failover scripts or created manually on the target.

  1. Mount DTFS on the directory on the source where the Samba share is located.
  2. Start the Double-Take daemon on the source.
  3. Stop and restart the Samba daemon on the source. The Double-Take daemon must be running before the Samba daemon in order for replication operations to be captured.
  4. On your target, set the SMB and WinBind daemons to manual startup. This allows the failover script to control when the daemons start on the target.
  5. Create a replication set on the source that includes /etc/SAMBA/samba_conf and the shared data that has DTFS mounted.
  6. Connect the replication set using the Connection Wizard or the Connection Manager.
  7. Add the following to your post-failover script.
  8. service smb start

  9. If necessary, update DNS.

After failover, the daemons will automatically be started by the post-failover script. If your clients see an access denied or share not found error message when attempting to access a share, they will need to remount the share.

 

Editing failover monitoring configuration

If you want to edit the monitor settings for a source that is currently being monitored, highlight that source on the Monitored Machines tree on the main Failover Control Center screen and click Edit. The Monitor Settings dialog box will open. Follow the instructions under Configuring failover monitoring.

Monitoring failover

Since it can be essential to quickly know the status of failover, Double-Take Availability offers various methods for monitoring the state of failover. When the Failover Control Center is running, you will see four visual indicators:

The following table identifies how the visual indicators change as the status of failover changes.

  Time to Fail Countdown Status Bar Colored Bullets Desktop Icon Tray
Source is Online

The Time to Fail counter is counting down and resetting each time a response is received from the source machine.

The status bar indicates that the target machine is monitoring the source machine.

The bullets are green.

When the Time to Fail value has decreased by 25% of the entire timeout period, the bullet changes from green to yellow, indicating that the target has not received a response from the source. The yellow bullet is a caution signal. If a response from the source is received, the countdown resets and the bullets change back to green. If the countdown reaches zero without the target receiving a response from the source, failover begins.

The Windows desktop icon tray contains a failover icon with red and green computers.
Source Fails and Failover is Initiated The Time to Fail countdown value is 0. The status bar displays the source machine and IP address currently being assumed by the target.

The bullets are red.

The Windows desktop icon tray contains a failover icon with red and green computers.

Failover is Complete The Time to Fail counter is replaced with a failed message.

The status bar indicates that monitoring has continued.

The bullets are red.

The Windows desktop icon tray contains a failover icon with a red computer.

Failing over

The failover process, including script processing, can be tested at any time. To force unavailability, disconnect the network cable from a monitored machine, wait for the Time to Fail counter to decrease to zero and failover begins. To avoid the countdown delay, highlight the monitored machine name in the Failover Control Center window and select Failover.

If Manual Intervention is enabled, the Failover Control Center will prompt you when a failure occurs.

Note:

If the Failover Control Center is not running at the time the failure occurs, the manual intervention dialog box will appear the next time the Failover Control Center is started.

When a failure occurs, an alert is forwarded to the Linux system log. You can then start the Failover Control Center and respond to the manual intervention prompt.

If SNMP is installed and configured, an SNMP trap is also generated. When using a third-party SNMP manager, an e-mail or page can be generated to notify you of the failure.

Click Cancel to abort the failover process. If necessary, you can initiate failover later from the Failover Control Center. Click OK to proceed with failover.

Failing back

When failover occurs, a source machine has failed. The steps below must be completed in order to complete failback without IP address or name conflicts.

  1. Verify that your source machine is not connected to the network. If it is, disconnect it.
  2. Resolve the source machine problem that caused the failure.
  3. Warning: Do not connect the source machine to the network at this time.
  4. Disconnect any replication sets that may have been connected from the original source. The source's replication sets will be identified with the restore required icon. Disconnect the source's replication sets by right-clicking on them and selecting Disconnect.
  5. In the Failover Control Center, select the source machine that is currently standing in for the failed source.
  6. Select the failed source and click Failback. If you specified a pre-failback script in your failover configuration, that script will be executed at this time.

  1. You will be prompted to determine if you want to continue monitoring the source. Bring the source online and select Continue or Stop to indicate if you want to continue monitoring the source.

After you have selected whether or not to continue monitoring the source machine, the source post-failback script, if configured, will be started.

At this time, the source machine must be online and Double-Take Availability must be running to ensure that the source post-failback script can be started. If the source has not completed its boot process, the command to start the script may be lost and the script will not be initiated.

Removing failover monitoring configuration

If you want to discontinue monitoring a source, highlight that machine on the Monitored Machines tree on the main Failover Control Center screen and click Remove Monitor. No additional dialog boxes will open.

Restoring Double-Take Availability data

The Replication Console provides an easy method for restoring replicated data from the target back to the original source or to a new source server. You are only required to input the original source, the target, and the name of the replication set you want to restore. Double-Take Availability handles the rest, including selecting the files in the replication set and restoring them to the correct location.

  1. Disconnect any replication sets that may have been connected from the original source. If your source failed over, the source's replication sets will be identified with the restore required icon. Even if you did not failover, you need to disconnect the original source replication sets before starting a restoration. Disconnect the source's replication sets by right-clicking on them and selecting Disconnect.
  2. From the Replication Console, select Tools, Restoration Manager.

  1. Identify the Original Source machine. This is your source machine where the data originally resided.
  2. Select the Restore From machine. This is the target machine where the copy of the data is stored.
  3. Replication Set contains the replication set information stored on the target machine (the machine in Restore From). If no replication sets are available, the list will be blank. Select the replication set that corresponds to the data that you need to restore.
  4. Note:

    Restoration applies to the entire replication set and should not be used for individual file restoration. Individual files can be restored manually by copying the files from the target to the source machine using native operating system tools.

  5. Select the Restore To machine. This is the machine where the copy of the data will be sent.
  6. The Restore To and Restore From paths will automatically be populated when the replication set is selected. The restore to path is the directory that is the common parent directory for all of the directories in the replication set. If the replication set crosses volumes, then there will be a separate path for each volume. The restore from path is the path on the target server where the replicated files are located.
  7. Note:

    Restoring across a NAT router requires the ports to be the same as the original connection. If the ports have been modified (manually or reinstalled), you must set the port numbers to the same values as the last valid source/target connection.

  8. Select the Use Backup Replication Set check box to use the target’s copy of the replication set database for the restoration. If this check box is not marked, you will be accessing the replication set information from the source.
  9. Select the Restore Replication Set check box to restore the target’s copy of the replication set database to the source during the restoration process.
  10. Select the restoration conditionals that you want to use.
  1. If you want to configure orphan files, click the Orphans tab. The same orphan options are available for a restoration connection as a standard connection.
  2. Click Restore to begin the restoration.

Server settings

Most of the Double-Take Availability server settings are located in the Replication Console Server Properties dialog box. To access this dialog box, right-click a server in the left pane of the Replication Console and select Properties. The Server Properties dialog box contains multiple tabs with the Double-Take Availability server settings. For information on the server settings not available through the Replication Console, see the Scripting Guide.

This section contains the following topics, each corresponding to a tab in the Server Properties dialog box.

Identifying a server

From the Replication Console, you can see server identity information, including a server's Double-Take Availability activation code.

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the General tab.

  1. Specify the server identity information. Some of the fields are informational only.
  1. Click OK to save the settings.

Licensing a server

From the Replication Console, you can manage your server activation codes. The activation code is the Double-Take Availability license which is required on every Double-Take Availability server. The activation code is a 24 character, alpha-numeric code. You can change your activation code without reinstalling, if your license changes. There are different licenses available.

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties.
  3. Select the Licensing tab. The fields displayed on this tab will vary depending on your activation code(s).

  1. Enter an activation code and click Add. Repeat for each activation code.
  2. Highlight an activation code in the list to display any status messages for that code below the list display.
  3. If you need to remove a code from the server, highlight it in the list and click Remove.
  4. To update a temporary node-locked license to a permanent license, you need to provide server information which will be used to generate a permanent node-locked license.
  1. Highlight your activation code in the list to display the Node-Locking section at the bottom of the Licensing tab.
  2. Click the hyperlink in the Node-Locking section. If you do not have an Internet connection, copy the Server Information text from the Node-Locking section into the form at https://activate.doubletake.com from another machine.
  3. After you submit the form, you will receive an email with a node-locked license key. Enter that key on the Licensing tab and click Add. The permanent activation code is specific to this server. It cannot be used on any other server. If the activation code and server do not match, Double-Take Availability will not run.
  1. Click OK to save the settings.

Configuring server startup options

From the Replication Console, you can configure server startup options for each Double-Take Availability server.

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Setup tab.

  1. Specify the server setup and source startup options.

Task command processing can be enabled from the Replication Console, but it can only be initiated through the scripting language. See the Scripting Guide for more information.

If you disable this option on a source server, you can still submit tasks to be processed on a target, although task command processing must be enabled on the target.

  1. Click OK to save the settings.

Configuring network communication properties for a server

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Network tab.

  1. Specify the network communication properties.
  1. Click OK to save the settings.

Queuing data

You should configure queuing on both the source and target.

  1. Right-click the server on the left pane of the Replication Console.
  2. Select Properties.
  3. Select the Queue tab.
  4. Specify the queue settings for the server.

Select a location on a volume that will have minimal impact on the operating system and applications being protected. For best results and reliability, this should be a dedicated, non-boot volume. The disk queue should not be on the same physical or logical volume as the data being replicated.

Note:

Scanning the Double-Take Availability queue files for viruses can cause unexpected results. If anti-virus software detects a virus in a queue file and deletes or moves it, data integrity on the target cannot be guaranteed. As long as you have your anti-virus software configured to protect the actual production data, the anti-virus software can clean, delete, or move an infected file and the clean, delete, or move will be replicated to the target. This will keep the target from becoming infected and will not impact the Double-Take Availability queues.

Since the source is typically running a production application, it is important that the amount of memory Double-Take Availability and the other applications use does not exceed the amount of RAM in the system. If the applications are configured to use more memory than there is RAM, the system will begin to swap pages of memory to disk and the system performance will degrade. For example, by default an application may be configured to use all of the available system memory when needed, and this may happen during high-load operations. These high-load operations cause Double-Take Availability to need memory to queue the data being changed by the application. In this case, you would need to configure the applications so that they collectively do not exceed the amount of RAM on the server. Perhaps on a server with 1 GB of RAM running the application and Double-Take Availability, you might configure the application to use 512 MB and Double-Take Availability to use 256 MB, leaving 256 MB for the operating system and other applications on the system. Many server applications default to using all available system memory, so it is important to check and configure applications appropriately, particularly on high-capacity servers.

Any changes to the memory usage will not take effect until the Double-Take daemon has been restarted on the server.

  1. Click OK to save the settings.

Configuring source data processing options

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Source tab.

  1. Specify how the source will process data.
  1. Click OK to save the settings.

Configuring target data processing options

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Target tab.

  1. Specify how the target will process data.
  1. Click OK to save the settings.

Specifying the Double-Take Availability database storage files

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Database tab.

  1. Specify the database files that store the Double-Take Availability replication set, connection, and scheduling information.
  1. Click OK to save the settings.

Specifying file names for logging and statistics

  1. Right-click a server on the left pane of the Replication Console.
  2. Select Properties
  3. Select the Logging tab.

  1. Specify the location and file names for the log and statistics files.
  1. Click OK to save the settings.

E-mailing system messages

You can e-mail system messages to specified addresses. The subject of the e-mail will contain an optional prefix, the server name where the message was logged, the message ID, and the severity level (information, warning, or error). The text of the message will be displayed in the body of the e-mail message.

  1. To enable e-mail notification for a server, right-click the server in the left pane of the Replication Console and select Properties.
  2. Select the E-mail Notification tab.

  1. Select Enable notification.
  2. Note: Any specified notification settings are retained when Enable notification is disabled.
  3. Specify your e-mail settings.

If desired, enter unique text for the Subject Prefix which will be inserted at the front of the subject line for each Double-Take Availability e-mail message. This will help distinguish Double-Take Availability messages from other messages. This field is optional.

If desired, enable Add event description to subject to have the description of the message appended to the end of the subject line. This field is optional.

Security

To ensure protection of your data, Double-Take Availability offers multi-level security using native operating system security features. Privileges are granted through membership in user groups defined on each machine running Double-Take Availability. To gain access to a particular Double-Take Availability source or target, the user must provide a valid operating system user name and password and the specified user name must be a member of one of the Double-Take Availability security groups. Once a valid user name and password have been provided and the Double-Take Availability source or target has verified membership in one of the Double-Take Availability security groups, the user is granted appropriate access to the source or target and the corresponding features are enabled in the client. Access to Double-Take Availability is granted on one of the following three levels.

The following table identifies which key Double-Take Availability features are available depending on the security access granted.

Double-Take Availability Feature Administrator Access

Monitor Access

View replication sets and rules X X
Modify replication sets and rules X  
View connections, mirroring, replication, verification, failover, and restoration X X
Control connections, mirroring, replication, verification, failover, and restoration X  
View server settings X X
Modify server settings X  
Note: Although Double-Take Availability passwords are encrypted when they are stored, Double-Take Availabilitysecurity design does assume that any machine running the Double-Take Availability client application is protected from unauthorized access. If you are running the Double-Take Availability client and step away from your machine, you must protect your machine from unauthorized access.

Logging on and off

To ensure protection of your data, Double-Take Availability offers multi-level security using native operating system security features. Privileges are granted through membership in user groups defined on each machine running Double-Take Availability. To gain access to a particular Double-Take Availability source or target, the user must provide a valid operating system user name and password and the specified user name must be a member of one of the Double-Take Availability security groups. Once a valid user name and password has been provided and the Double-Take Availability source or target has verified membership in one of the Double-Take Availability security groups, the user is granted appropriate access to the source or target and the corresponding features are enabled in the client. Access to Double-Take Availability is granted on one of the following three levels.

  1. Highlight a machine on the left pane of the Replication Console. By double-clicking the machine name, Double-Take Availability automatically attempts to log you on to the selected machine using the ID that you are currently logged on with. Verify your access by the resulting icon.
  2. If you have no access, the Logon dialog box will automatically appear. If you have monitor access or want to log on with a different username, right-click the machine name and select Logon.
  1. Specify your Username and Password and whether you want your password saved.

  1. Click OK. Verify your access by the resulting icon and log on again if necessary.
  2. Icon Description Access Granted
    This icon is a computer with a gear and it indicates the Double-Take Availability security is set to administrator access. Administrator rights
    This icon is a computer with a magnifying glass and it indicates the Double-Take Availability security is set to monitor only access. Monitor rights
    This icon is a lock and it indicates the Double-Take Availability security is set to no access. No rights
    Note:

    When logging in, the user name, password, and domain are limited to 100 characters. If your activation code is missing or invalid, you will be prompted to open the Server Properties General tab to add or correct the code. Select Yes to open the Server Properties dialog box or select No to continue without adding an activation code.

    If the login does not complete within 30 seconds, it is automatically canceled. If this timeout is not long enough for your environment, you can increase it by adjusting the Communication Timeout on the Configuration tab of the Replication Console properties. Select File, Options, from the Replication Console to access this screen.

    Double-Take Availability uses ICMP pings to verify server availability during the login process. If your Double-Take Availability server is across a router or firewall that has ICMP pings disabled, you will need to disable the Double-Take Availability ICMP ping verification. To do this, select File, Options, from the Replication Console and disable Use ICMP to verify server availability.

  3. To log off of a Double-Take Availability machine, right-click the machine name on the left pane of the Replication Console and select Logout.

Evaluating Double-Take Availability

The following evaluation procedure has eleven tasks containing step-by-step instructions for evaluating the core functionality of Double-Take Availability, specifically mirroring, replication, failover, and restoration. This is a good process for users who want to see, first-hand, the benefits that Double-Take Availabilityhas to offer.

Before starting this evaluation procedure, make sure you have Double-Take Availability installed on the source and target. You should have at least 1 GB of data on the source for testing. If you are going to be protecting application data, make sure the application is pre-installed on the target, but the application is not running on the target. If the application is running on the target, the files will be held open and Double-Take Availability will not be able to write to the files. In the even of a source failure, the application can be started on the target and the files can then be accessed.

This evaluation consists of the following tasks.

1. Establishing a connection
  1. Start the Double-Take Availability Replication Console by selecting Start, Programs, Double-Take, Replication Console.
  2. Click Make a connection from the right pane of the Replication Console. If that quick launch screen is no longer visible, select Tools, Connection Wizard.
  3. Note:

    If the Double-Take Servers root is highlighted in the left pane of the Replication Console, the Connection Wizard menu option will not be available. To access the menu, expand the server tree in the left pane, and highlight a server in the tree.

  1. The Connection Wizard opens to the Welcome screen. Review this screen and click Next to continue.
  2. Note:

    At any time while using the Connection Wizard, click Back to return to previous screens and review your selections.

  1. If you highlighted a source in the Replication Console, the source will already be selected. If it is not, select the Double-Take Availability source. This is the server that you want to protect. Click Next to continue.
  2. Note:

    Double-Take Availability will automatically attempt to log on to the selected source using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. If you highlighted a target in the Replication Console, the target will already be selected. If it is not, select the Double-Take Availability target. This is your backup server that will protect the source. Click Next to continue.
  2. Note:

    Double-Take Availability will automatically attempt to log on to the selected target using the identification of the user logged on to the local machine. If the logon is not successful, the Logon dialog box will appear prompting for your security identification. When logging in, the user name, password, and domain are limited to 100 characters.

  1. On the next screen, verify Create a new replication set with this name is selected.
  2. Enter a name for your replication set, and click Next to continue.
  3. A tree display appears identifying the volumes and directories available on your source. Mark the check box of the volumes and/or directories you want to protect. Click Next to continue.
  4. There are two pre-defined locations to store the source data on the target, or you can select a custom location. For this evaluation, select the option Send all data to the same path on the target. This option keeps the directory structure on the source and target identical. For example, /var/data is transmitted to /var/data. Click Next to continue.
  5. Review your selections on the summary screen. You do not need to set any advanced options for this evaluation, so click Finish. The Connection Wizard will close, the connection will be established, and mirroring and replication will begin.
  6. You will be prompted to save your newly created replication set. Click Yes to save it.
2. Monitoring the activity and completion of the initial mirror

View your new connection in the Replication Console by highlighting the source on the left pane. The connection will appear on the right pane. Use the horizontal scroll bar at the bottom of the right pane to view the various status columns. Pay attention to the Mirror Status column which shows the status of the mirroring operation. During the mirroring process, you will see a percentage of the mirror that has been completed. When the Mirror Status changes to Idle, there is no mirroring activity, meaning your initial mirror has completed.

To view specific mirroring statistics that may be of interest, use the horizontal scroll bar at the bottom of the right pane of the Replication Console window to view the various columns.

Monitoring a connection through the Replication Console contains complete details on all of the Replication Console statistics.

After your mirror is complete, look at your target and you will see the replicated data stored in the location you specified. Now you are ready to continue with the evaluation.

3. Changing data to cause replication

In order to test replication, you need to change the data on your source. This includes modifying existing files, creating new files, deleting files, and changing permissions and attributes.

  1. On the source, browse through the directories and files contained in your replication set.
  2. Select four files from your source and record the file name, date, time, and file size for each file.
  3. On your target, locate those same four files that you just identified on your source. The files on the target match the files on the source.
  4. Back on your source, view the contents of one of your files contained in your replication set and note the file contents.
  5. On your target, view that same file that you just viewed on the source. The file contents on the target match the file contents on the source.
  6. Highlight your source in the left pane of the Replication Console.
  7. Locate the Replication Status and Sent (Bytes) columns in the right pane.
  8. Tile your Replication Console so that you can see it while still having access to your desktop.
  9. On your source, edit the file that you viewed above. Save your changes, and watch the Replication Console statistics as the file change causes replication to occur.
  10. Modify the other three files so that the date, time, and/or size is updated, and again watch the Replication Console statistics as the file changes cause replication to occur. While Double-Take Availability is actively replicating, the status will be Replicating. When there is no replication activity, the status is Ready.
  11. Use the horizontal scroll bars to display additional replication statistics.

Monitoring a connection through the Replication Console contains complete details on all of the Replication Console statistics.

Note:

Many user applications typically save an entire file even though only a portion of the file may have changed. Therefore, the replication statistics will show the entire file being transmitted, not just the changed data. To confirm that replication only transmits the changed segments of files, you must use an application, such as a database application, or a command, such as the echo command, to save only the changed portions of a file.

You may notice your Replication Status toggle between Replicating and Ready as it continues processing the file changes, when your Replication Status stays at Ready, Double-Take Availability is waiting for additional changes to transmit. After replication is complete, you are ready to continue with the evaluation.

4. Verifying the data changes on the target

Now that you have modified some of the files, you want to be sure that the file modifications were applied correctly.

Note:

Machines that are doing minimal or light processing, as you are in this evaluation, may have file operations that remain in memory until additional operations flush them out and they are written to disk. This may make Double-Take Availability files on the target appear as if they are not synchronized. When the operations in memory are released, the files will be updated on the target. To make sure this does not impact your testing, flush memory by copying a couple of files from one directory to another and then deleting them. You can also use the sync program (which exercises the sync(2) system call) to flush memory.

  1. Browse your source and target. Compare the directory structures and the total number of files.
  2. Look again at the four files you modified earlier. Verify manually that the changes you made have been applied to the target copy of the file.
  3. Right-click the connection on the right pane of the Replication Console and select Verify. You will see two choices on the Start Verification dialog box.
  1. Select Verify only and click OK.

Just like when you were monitoring the mirror and replication processes, you can monitor the verification process. Notice that the Mirror Status column changes to Verifying while the verification process occurs. When the verification is complete, Double-Take Availability will have created a log file for you to review.

  1. Wait until your Mirror Status has returned to Idle and then open the file DTVerify.log located in the Double-Take Availability installation directory on your source. You will see that all of the files are reported as the same.
  2. Modify one of your files on the target and repeat the verification process, but this time, select Remirror data to the target automatically.
  3. Note:

    Since your target file is newer, make sure that Only if the source’s date is newer than the target’s date is not selected.

  4. Look at the file on the target that you modified and confirm that your changes are gone. The source version has overwritten the file on the target.
5. Testing your target data

At this point in your evaluation, you may want to test your target data. The type of testing you will need to perform will depend on the type of data you are protecting.

  1. In order to test the application data on the target, you will need to start the application on the target. But Double-Take Availability requires applications to be in a standby mode in order to update files on the target. In order to meet both of those requirements, you will need to pause the target. When you pause the target, the source begins to queue the data changes that are occurring. This will give you an opportunity to start the services on the target, test the data, stop the services, and then resume the target. Make sure your mirror is Idle and then pause the target by right-clicking the connection in the Replication Console and selecting Pause Target.
  2. Once the target is paused, you can start the application services on the target. Test the application data by using clients to connect to the application. For this evaluation, the clients will need to be configured to access the application from the target. In a real-world scenario, if failover has occurred, the target would be standing in for the source and the clients would still be accessing the application from the source identity.
  3. After you have completed your testing, stop the application services on the target.
  4. After the application services on the target have been stopped, you can resume your target through the Replication Console by right-clicking the connection and selecting Resume Target.
  5. While you were testing the application on the target, the application files were updated on the target, thus your source and target are no longer synchronized. You will need to perform a manual remirror to resynchronize the files on the source and target. Right-click the connection and select Mirroring, Start. Select a Difference Mirror. Make sure that Only send if source’s date is newer than the target’s date is not selected. Since your target files are actually newer than the source (because of the testing you performed), you do want the newer files on the target to be overwritten by the files from the source. Click OK to begin the mirror.

When the mirror is complete, your source and target will again be synchronized and you can continue with your evaluation.

6. Configuring failover monitoring

The following instructions will configure failover monitoring.

  1. The Failover Control Center can be started from within the Replication Console or from the Windows desktop.
  1. Select your target from the Target Machine list box.
  2. Click Login to login to the selected target.
  3. Click Add Monitor. The Insert Source Machine dialog box appears in front of the Monitor Settings dialog box.
  4. Type in your source machine name and click OK. The Insert Source Machine dialog box will close and the Monitor Settings dialog box will be available for updating. This is where you configure failover monitoring.
  5. Select the source to be monitored by marking the check box to the left of the source server name in the Names to Monitor tree.

At this point, in terms of your evaluation, your failover configuration is complete because you will be using the default settings for the remaining options. But while you are viewing the Monitor Settings dialog box, notice the configuration options available to you.

7. Monitoring failover

Since it can be essential to quickly know the status of failover, Double-Take Availability offers various methods for monitoring the state of failover. When the Failover Control Center is running, you will see four visual indicators.

Note:

You can minimize the Failover Control Center and, although it will not appear in your Windows taskbar, it will still be active and the failover icon will still appear in the desktop icon tray.

The Failover Control Center does not have to be running for failover to occur.

Monitoring failover contains more information on the Failover Control Center visual indicators.

8. Simulating a failure

To fully evaluate failover, you need to simulate a failure. The Failover Control Center does not have to be running in order for failover to occur, but for the purpose of this evaluation, make sure that it is running so that you can see each step of the process.

  1. Ping the source’s IP address from a client machine.
  2. Ping the source’s machine name from a client machine.
  3. Disconnect the network cable(s) on the source. Notice immediately, that the Failover Control Center Time to Fail counter decreases and never resets. You will see the icons change to yellow and eventually to red. Once the icons are red and the Failed Over message is displayed, failover has occurred.
  4. Note:

    The Linux system log on the target provides details on the actual steps that have occurred during failover.

  1. Ping the source’s IP address from a client machine.
  2. Ping the source’s machine name from a client machine.

As you can see, the target has taken on the identity of the source. Application and user requests destined for the source are routed directly to the target. The impact on your end users is minimal.

9. Simulating data changes after failover

While your source is failed over to your target, end users continue to work without interruption and the data on the target will be updated. To fully evaluate the next step, restoration, simulate the changes that the end users would have made on the target while the source was unavailable.

  1. Identify the file that you edited earlier on the source.
  2. Locate that same file on the target and make edits to it. Save the changes.
  3. Repeat that process, modifying the other three files from earlier, but this time make the modifications on the target copy of the file. Save the changes.

If desired, you can also test the target data as you did earlier. You can test user data using the associated application, and you can save the changes if desired. If you want to test application data, start the application services on the target, and test the application data by using clients to connect to the application. Because the source is now failed over, you will not need to worry about pausing the target or configuring clients to access the application from the target. The clients will continue to access the source, which is now being handled by the target machine.

10. Initiating failback

When failover occurs, a source machine has failed. The steps below must be used to complete failback, which releases the source identity from the target. .

  1. If this were a real failure scenario and not an evaluation, you would first verify that your source machine is not connected to the network. If it is, you would have to disconnect it from the network.
  2. Next you would resolve the source machine problem that caused the failure.
  3. Warning: Do not connect the source machine to the network at this time.
  4. In the Failover Control Center, select the target that is currently standing in for the failed source.
  5. Select the failed source and click Failback.

  1. You will be prompted to determine if you want to continue monitoring the source. Do not make any selections at this time.
  2. At this time, you would connect the source to the network. For this evaluation, reconnect the network cable(s) on the source that you disconnected to simulate the failure.
  3. After the source is online, select Stop in the Failover Control Center to indicate that you do not want to continue monitoring the source.

At this time, your target is back to its original identity and the source is back online.

11. Restoring your data

The Replication Console provides an easy method for restoring replicated data from the target back to the original source or to a new source server. You are only required to input the original source, the target, and the name of the replication set you want to restore. Double-Take Availability handles the rest, including selecting the files in the replication set and restoring them to the correct location.

  1. From the Replication Console, select Tools, Restoration Manager.

  1. Identify the Original Source machine. This is your source machine where the data originally resided.
  2. Select the Restore From machine. This is the target machine where the copy of the data is stored.
  3. Replication Set contains the replication set information stored on the target machine (the machine in Restore From). If no replication sets are available, the list will be blank. Select the replication set that corresponds to the data that you need to restore.
  4. Select the Restore To machine. For this evaluation, select the original source. This is the machine where the copy of the data will be sent.
  5. The Restore To and Restore From paths will automatically be populated when the replication set is selected. The restore to path is the directory that is the common parent directory for all of the directories in the replication set. If the replication set crosses volumes, then there will be a separate path for each volume. The restore from path is the path on the target server where the replicated files are located.
  6. Use the default settings for the remaining restoration options.
  7. Click Restore to begin the restoration.

Once the restoration is complete, your evaluation is complete. Congratulations!